Dorian Fevrier's blog - Mot-clé - slowdownJe m’appelle FEVRIER Dorian, je suis infographiste 3D passionné par mon métier, l’informatique en général, l’internet, la programmation et l’évolution de tout ce petit monde. Vous trouverez sur ce blog des tutoriaux, mes coups de cœurs, avis, etc.2024-01-02T23:48:05+01:00FEVRIER Dorianurn:md5:695d9c73474c33ce3dab043823509c4bDotclearResolve network slowdowns due to Nuke, After Effects and others software like them (English Translation)urn:md5:83166070f4695a715ca359133a4fb7a02012-10-19T10:31:00+02:002014-11-26T14:51:19+01:00NarannScript et codeafter effectennetworknukepythonslowdown<p><img src="https://www.fevrierdorian.com/blog/public/billets/2012_10_14_nuke_reseau_lenteurs/nuke_reseau_tn.png" alt="nuke_reseau_tn.png" style="float:left; margin: 0 1em 1em 0;" title="nuke_reseau_tn.png, oct. 2012" height="150" width="150" />This is an english translation of a <a href="https://www.fevrierdorian.com/blog/post/2012/10/14/Resoudre-les-ralentissements-reseaux-dus-a-Nuke-After-Effect-et-autres-logiciels-du-genre">ticket I wrote</a> (in french :franceHappy: ) a few days ago.</p>
<p>Nuke, After Effects and probably other softwares, can be very "greedy" in terms of disk accesses. So much so they can break network performances drastically if several stations are rendering.</p>
<p>In this ticket I will show you a quick explanation of the why and how a small Python class that can be used as a prototype to solve the problem.</p> <h3>What's the problem?</h3>
<p>Let's start with a concrete exemple:</p>
<p>At the end of <a href="http://www.imdb.com/title/tt1482459/" hreflang="en">Lorax</a>, many compositors were rendering their images all at the same time and time estimations given by Nuke were sometime surprising (3h-4h left for a simple nuke script). Monitoring a file being written, I realized its weight was increasing very slowly. It was 100ko, then 300ko, then after 10 minutes it became 400ko etc... I concluded the network was overloaded... :reflexionIntense:</p>
<p>I remembered that we had the same worries on <a href="http://www.allocine.fr/film/fichefilm_gen_cfilm=183772.html" hreflang="fr">Tales Of The Night</a>. Our infrastructure was certainly much smaller but 3 After Effect renders were "pumping" the entire network. The found solution at that time was to render the file in local and copy it once finished. The effects were immediate: No more slowdown with the network. :sauteJoie:</p>
<p>So I tried to render the nuke script of a CG artist locally to see if it reduces the problem. The largest files to read were source files (because they are many), and the final image weight was actually very light so I was not thinking it could change anything. But, once again, the conclusion was clear: The rendering was finished in 10-15 minutes (I'm not kidding) instead of 3-4hrs... :laClasse:</p>
<p>I thought it was maybe the writing process which was a problem. However I tried to the copy the new images sequence to the server, and it happened to be very fast. So I've done this with every artist, one by one, then the network has no bottleneck anymore and every renders completed.</p>
<p>But this was a very poor fix. We had to understand why Nuke couldn't write its images faster through the network. After <a href="http://forums.thefoundry.co.uk/phpBB2/viewtopic.php?p=29117" hreflang="en">having asked</a> on the nuke-users mailing list (where I could see I was not alone but The Foundry didn't seem to be able to fix this), I've started to "profile" Nuke to know how it does the job (<a href="http://en.wikipedia.org/wiki/Strace" hreflang="en">strace</a> and <a href="http://linux.die.net/man/1/inotifywatch" hreflang="en">inotifywatch</a> are you friends :dentcasse: ).</p>
<p>Sometimes conclusion seems to be obvious but it's always good to check you assumptions:</p>
<p>With Nuke, if you write a <em>zip 1 line</em> compressed EXR, in 1920x1080, Nuke will do a little less than 900 (approximately) write accesses on the file. If you are in <em>zip 16 lines</em>, it will do about 70 accesses (1080/16). And in uncompressed, that's really 1080 accesses. :trollface:</p>
<p>In facts, compress in <em>zip 16 line</em> is not efficient if images should be read by Nuke. And depending of your network infrastructure, write line by line can put it completely down. It's difficult to explain how finally few Nuke rendering can fill a network, even if this one is strong. I feel this is related to multithreading: Nuke reads images (often, many at the same time) on the network while it is writing through it.</p>
<p>The most obvious solution is therefore to write the rendered image(s) on the local disk and to copy it in one time (one access) on the network disk. If you don't have technical resources (or just time), it's the simplest approach, but on larger projects it can quickly become daunting and (lest we forget) source of errors.</p>
<p>There are several solutions and I was leaning on a prototype that I found interesting because it's easy to implement.</p>
<p><img src="https://www.fevrierdorian.com/blog/public/billets/2012_10_14_nuke_reseau_lenteurs/problems.jpg" alt="problems.jpg" style="display:block; margin:0 auto;" title="problems.jpg, oct. 2012" height="440" width="720" /></p>
<h3>The principle</h3>
<ul>
<li>You launch a Python thread that will watch a folder.</li>
<li>Every three seconds, the thread will list present files in a folder and check if their name will match to a given regular expression (the "pattern" of your file name).</li>
<li>If the file seems to be a file you want to move once finished, it searches the same file with ".finished" (example: "toto.001.exr.finished").</li>
<li>If this file exists, it moves the origin file and removes the ".finished" one and start the main loop again.</li>
<li>Once the render is done (so every images are finished), you ask the thread to stop itself.</li>
</ul>
<p>As you can see, this method requires that you create a ".finished" file each time an image is finished. This is because it's impossible for the thread to know when an image is actually completed. The creation of this ".finished" file can be handled in a thousand different ways (For Maya, a simple "Post render frame" should do the job) so I won't go into the details. :siffle:</p>
<h3>The code</h3>
<p>Here it is:</p>
<pre class="python python"><span style="color: #ff7700;font-weight:bold;">import</span> <span style="color: #dc143c;">os</span>, <span style="color: #dc143c;">threading</span>, <span style="color: #dc143c;">re</span>, <span style="color: #dc143c;">time</span>
<span style="color: #ff7700;font-weight:bold;">class</span> MoverThread<span style="color: black;">(</span> <span style="color: #dc143c;">threading</span>.<span style="color: black;">Thread</span> <span style="color: black;">)</span> :
<span style="color: #ff7700;font-weight:bold;">def</span> <span style="color: #0000cd;">__init__</span><span style="color: black;">(</span> <span style="color: #008000;">self</span>, dirTocheck, dirToMoveIn, patternToCheck, force=<span style="color: #008000;">False</span> <span style="color: black;">)</span> :
<span style="color: #dc143c;">threading</span>.<span style="color: black;">Thread</span>.<span style="color: #0000cd;">__init__</span><span style="color: black;">(</span> <span style="color: #008000;">self</span> <span style="color: black;">)</span>
<span style="color: #008000;">self</span>._terminate = <span style="color: #008000;">False</span>
<span style="color: #008000;">self</span>.<span style="color: black;">dirTocheck</span> = dirTocheck
<span style="color: #008000;">self</span>.<span style="color: black;">dirToMoveIn</span> = dirToMoveIn
<span style="color: #008000;">self</span>.<span style="color: black;">force</span> = force
<span style="color: #808080; font-style: italic;"># regex pattern</span>
<span style="color: #008000;">self</span>.<span style="color: black;">patternToCheck</span> = patternToCheck
<span style="color: #008000;">self</span>.<span style="color: black;">rePattern</span> = <span style="color: #dc143c;">re</span>.<span style="color: #008000;">compile</span><span style="color: black;">(</span> patternToCheck <span style="color: black;">)</span>
<span style="color: #808080; font-style: italic;"># sanity check</span>
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #dc143c;">os</span>.<span style="color: black;">path</span>.<span style="color: black;">isdir</span><span style="color: black;">(</span><span style="color: #008000;">self</span>.<span style="color: black;">dirTocheck</span><span style="color: black;">)</span> :
<span style="color: #ff7700;font-weight:bold;">raise</span> <span style="color: #008000;">Exception</span><span style="color: black;">(</span> <span style="color: #483d8b;">"The given directory (dirTocheck) is not a valid directory -> %s"</span> <span style="color: #66cc66;">%</span> <span style="color: #008000;">self</span>.<span style="color: black;">dirTocheck</span> <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #dc143c;">os</span>.<span style="color: black;">path</span>.<span style="color: black;">isdir</span><span style="color: black;">(</span><span style="color: #008000;">self</span>.<span style="color: black;">dirToMoveIn</span><span style="color: black;">)</span> :
<span style="color: #ff7700;font-weight:bold;">raise</span> <span style="color: #008000;">Exception</span><span style="color: black;">(</span> <span style="color: #483d8b;">"The given directory (dirToMoveIn) is not a valid directory -> %s"</span> <span style="color: #66cc66;">%</span> <span style="color: #008000;">self</span>.<span style="color: black;">dirToMoveIn</span> <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">def</span> run<span style="color: black;">(</span> <span style="color: #008000;">self</span> <span style="color: black;">)</span> :
filesNotMoved = <span style="color: black;">[</span><span style="color: black;">]</span>
<span style="color: #ff7700;font-weight:bold;">while</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #008000;">self</span>._terminate :
<span style="color: #808080; font-style: italic;"># we wait 3 seconds before do anything</span>
<span style="color: #dc143c;">time</span>.<span style="color: black;">sleep</span><span style="color: black;">(</span> <span style="color: #ff4500;">3</span> <span style="color: black;">)</span>
<span style="color: #808080; font-style: italic;"># for every "entry" (file or folder) in the folder we check it have the good pattern. If it has, we check for a ".finished" file</span>
<span style="color: #ff7700;font-weight:bold;">for</span> entry <span style="color: #ff7700;font-weight:bold;">in</span> <span style="color: #dc143c;">os</span>.<span style="color: black;">listdir</span><span style="color: black;">(</span> <span style="color: #008000;">self</span>.<span style="color: black;">dirTocheck</span> <span style="color: black;">)</span> :
<span style="color: #808080; font-style: italic;"># check the current entry is "compliant" with the given regex</span>
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #008000;">self</span>.<span style="color: black;">rePattern</span>.<span style="color: black;">match</span><span style="color: black;">(</span> entry <span style="color: black;">)</span> :
<span style="color: #ff7700;font-weight:bold;">continue</span>
srcFilePath = <span style="color: #dc143c;">os</span>.<span style="color: black;">path</span>.<span style="color: black;">join</span><span style="color: black;">(</span> <span style="color: #008000;">self</span>.<span style="color: black;">dirTocheck</span>, entry <span style="color: black;">)</span>
dstFilePath = <span style="color: #dc143c;">os</span>.<span style="color: black;">path</span>.<span style="color: black;">join</span><span style="color: black;">(</span> <span style="color: #008000;">self</span>.<span style="color: black;">dirToMoveIn</span>, entry <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #dc143c;">os</span>.<span style="color: black;">path</span>.<span style="color: black;">isfile</span><span style="color: black;">(</span> srcFilePath+<span style="color: #483d8b;">".finished"</span> <span style="color: black;">)</span> :
<span style="color: #808080; font-style: italic;"># destination file aready exist?</span>
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #dc143c;">os</span>.<span style="color: black;">path</span>.<span style="color: black;">isfile</span><span style="color: black;">(</span> dstFilePath <span style="color: black;">)</span> <span style="color: #ff7700;font-weight:bold;">and</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #008000;">self</span>.<span style="color: black;">force</span>:
<span style="color: #808080; font-style: italic;"># don't add the entry if it is already in the list</span>
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #ff7700;font-weight:bold;">not</span> entry <span style="color: #ff7700;font-weight:bold;">in</span> filesNotMoved :
filesNotMoved.<span style="color: black;">append</span><span style="color: black;">(</span> entry <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">continue</span>
<span style="color: #808080; font-style: italic;"># move the file to it new location</span>
<span style="color: #dc143c;">os</span>.<span style="color: black;">rename</span><span style="color: black;">(</span> srcFilePath, dstFilePath <span style="color: black;">)</span>
<span style="color: #dc143c;">os</span>.<span style="color: black;">remove</span><span style="color: black;">(</span> srcFilePath+<span style="color: #483d8b;">".finished"</span> <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">print</span> <span style="color: #483d8b;">"File %s moved to %s"</span> <span style="color: #66cc66;">%</span> <span style="color: black;">(</span> entry, <span style="color: #008000;">self</span>.<span style="color: black;">dirToMoveIn</span> <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">break</span> <span style="color: #808080; font-style: italic;"># restart the while loop to avoid to continue the list of file we maybe have removed: ".finished"</span>
<span style="color: #ff7700;font-weight:bold;">print</span> <span style="color: #483d8b;">"Terminated!"</span>
<span style="color: #ff7700;font-weight:bold;">for</span> fileNotMoved <span style="color: #ff7700;font-weight:bold;">in</span> filesNotMoved :
<span style="color: #ff7700;font-weight:bold;">print</span> <span style="color: #483d8b;">"Already exists: Can't move %s to %s"</span> <span style="color: #66cc66;">%</span> <span style="color: black;">(</span> fileNotMoved, <span style="color: #008000;">self</span>.<span style="color: black;">dirToMoveIn</span> <span style="color: black;">)</span>
<span style="color: #ff7700;font-weight:bold;">def</span> join<span style="color: black;">(</span> <span style="color: #008000;">self</span> <span style="color: black;">)</span> :
<span style="color: #008000;">self</span>._terminate = <span style="color: #008000;">True</span>
<span style="color: #dc143c;">threading</span>.<span style="color: black;">Thread</span>.<span style="color: black;">join</span><span style="color: black;">(</span> <span style="color: #008000;">self</span> <span style="color: black;">)</span></pre>
<p>As you can see (or not), everything happens in a thread.</p>
<p>It's used like this:</p>
<pre class="python python"><span style="color: #ff7700;font-weight:bold;">import</span> waitFinishAndCopy
myMoverThread = waitFinishAndCopy.<span style="color: black;">MoverThread</span><span style="color: black;">(</span><span style="color: #483d8b;">"/a/local/path/"</span>, <span style="color: #483d8b;">"/a/network/path/"</span>, <span style="color: #483d8b;">"^toto<span style="color: #000099; font-weight: bold;">\.</span>[0-9]{4}<span style="color: #000099; font-weight: bold;">\.</span>exr$"</span><span style="color: black;">)</span>
myMoverThread.<span style="color: black;">start</span><span style="color: black;">(</span><span style="color: black;">)</span>
<span style="color: #808080; font-style: italic;"># start rendering, do rendering, end rendering.</span>
myMoverThread.<span style="color: black;">join</span><span style="color: black;">(</span><span style="color: black;">)</span></pre>
<p>And voila!</p>
<h3>Conclusion</h3>
<p>I hope this modest prototype will inspire you if you are experiencing delays on your network. :mechantCrash:</p>
<p>I also suggest to do some profiling on your core network applications, especially if they are used by many people. Their behavior is always interesting (and sometimes surprising).</p>
<p>Have a nice day!</p>
<p>Dorian</p>
<center>:marioCours:</center>