My LCD screen is starting to return to a less blue shade after enduring a stream of classic Anglo-Saxon last night. The cause of this was the crashing of the GUI software with an Out of Memory error about 98% of the way through the biggest part of the RepRap - the carriage.
I knew things were not going well. Garbage collects were happening with increasing frequency. I'd run the job to a nullcartesian device first to make sure it would print - probably best to close and restart given the current state of play.
I had "top" running, so I know it was all Java. No other applications of any consequence going on a 2GB machine.
Now the good news: Though technically incomplete, on inspection the part appears quite functional. There are some big blobs on it where the extruder was left running in garbage collects (should we force a GC at the end of a layer? Can we, or will Java "know better"?), but these will soon succumb to my trusty Dremmel tool. I believe I can now start work on the X axis assembly, the first phase of mechanical construction in our instructions. Yeah, looks like the picture on the left :) I leant on the X motor bracket and cracked it, but clear epoxy cures all ills.Vik :v)
I ran into a similar problem running the simulator on one of the parts. Kept getting slower and slower until it finally gave up. I bumped up the heap space parameter (-Xmx) and it ran much smoother. Worth a shot.
ReplyDeleteIf I give her any more she'll start swapping to disk. That way madness lies...
ReplyDeleteVik :v)
Why does it need so much RAM? It doesn't make sense. Presuming it rasterizes one layer at a time, the storage requirements should be pretty reasonable. Sounds a lot like a bug to me.
ReplyDeleteI suspect a bug in Java 3D. However, the air is already blue again - I just leant on the X motor bracket and cracked it.
ReplyDeleteArrrghhh!
Where clear epoxy gone?
Vik :v)
Sounds like there are some references kept to the already printed layers, causing Java to keep them in memory (running the garbage collector won't help in this case). Only the current layer would be needed, unless you want to show some kind of live preview of the piece containing each printed layer (I haven't looked at the host software, so maybe it is doing that?).
ReplyDeleteSo when do you expect to have a full set of Darwin parts, Vik? You seem to be getting damned close. :-)
ReplyDeleteThere's at least 40 solid hours of printing to go before I've got all the major parts, and another 20 hrs worth of diagonal brackets. I'm forbidden from running it overnight due to noise, and I have to go to Wellington this week, so I'm probably 3 weeks off.
ReplyDeleteVik :v)
So we're talking about 60 hours of printing remaining. How many hours have you spent printing up to this point.
ReplyDeleteAlso, what sort of print speed are you getting in mm/sec with Darwin.
Finally, you mention that this thing is too noisy to run at night. How noisy is noisy and are you afraid of it running amok while you are asleep as well?
I'm asking all this because I promised the French webzine Agoravox for a followup article about 16-18 months ago to the piece I did then. It's high time I told them where you and reprap have gone since then. They were very excited about the promise of Reprap. It will be fun to tell them that the promise is coming true now. :-D
I just had an idea about increasing the safety of leaving the machine running unattended, at night for example. Supposing you mounted a camera overlooking the deposition area, and calibrated it with a dry run so that, for a particular head location, you'd know which part of the image in which to expect movement during deposition. In the event that something drastic went wrong, for example the object under construction breaking loose from the bed or the build up of a big globule of plastic around the head, etc, subsequent motion of the head would would cause motion in the image outside the head's expected motion boundary, immediately indicating something had gone wrong and triggering an alarm (in the worst case scenario, smoke and flames would by default also register as unexpected motion ;-). Perhaps such an arrangement could be expanded in the future with proper machine vision in order to compare the workpiece with a rendering of its mesh as it should be , for example, but such a basic first step as mentioned above could probably be done relatively simply.
ReplyDeleteHere's a Java question. Can the memory management system spot that, if A points to B and B points to A, but nothing else points to either, both A and B can be junked?
ReplyDeleteIf it just uses reference counting it won't. If it does something classier, it will.
The geometry code uses lazy evaluation for the set-theoretic complements of objects, and (when a complement has been evaluated) A ia a thing and B the complement of the thing. If they never die, that could be what's using up all the memory...
In the Java world, that's called "data cancer". As long as an set object have some chain of references that loops back on itself, the garbage-collector will leave them alone, even if no thread has access to them. Your options are a) avoid circular references or b) create and call methods that explicitly break the chain.
ReplyDeleteerm, that should've been "as long as a set of objects"
ReplyDeleteJava expert (self proclaimed) to the rescue. :)
ReplyDeleteA couple of things:
1. You can't force a GC in Java, you can only suggest very nicely to the VM that it might like to have a go at the GC now. You can help though, buy putting your threads to sleep, an idle VM tends to keep busy with housekeeping.
2. Adrian, as regards your reference counting question, no the GC routines are cleverer than that (at least in all the versions of the JVM for the last few years), the nodes on the object graph need to be navigable to escape being garbage collected, non-navigable but circular links will be cleaned up.
Using the VisualVM you should be able to see a lot more of what is going on.
And a final question, where is this code residing, if it would help, I'd be more than happy to have a run though it with my profiling tools and see if I can spot any problems.
I tried using TPTP on it but the thing bogged down and didn't really gather any useful info, other than that it appeared to be creating a huge number of objects, mostly Rr2Point. On a very small, simple object, there were over 15000.
ReplyDeleteSounds like an optimal place to start the optimisation strategy has been found.
ReplyDeleteAnother way to avoid unreferenced objects is to copy new values into old objects, instead of creating new objects.
ReplyDeleteSo for example, you could have a Vector unreferencedRr2Points and a function getNewRr2Point(). Whenever a layer goes out of scope, add all the Rr2Points in that layer to unreferencedRr2Points. The function getUsedRr2Point() would get and remove the last Rr2Point from unreferencedRr2Points if the length of unreferencedRr2Points was greater than zero, otherwise it would create and return a new Rr2Point. You would then set the fields on the used Rr2Point and proceed as if you had created a new one.
This would eliminate the memory problem and might speed up the program, because creating java objects is slow.
(Assuming you're using the HotSpot JVM...)
ReplyDeleteAs a self-proclaimed Java GC expert [:-)], I can pass on the advice that you shouldn't worry about trying to be clever about memory management in Java. Objects that are created and immediately disposed are incredibly quick in Java these days. If you're having issues with full GCs causing pauses, look into one of the several low-pause collectors available (try -XX:+UseConcMarkSweepGC and -XX:+UseParNewGC). For more info, see http://java.sun.com/docs/hotspot/gc1.4.2/) Use the -XX:-PrintGCDetails flag to know how much time you're spending in GC.
Frequently, if you're spending a significant amount of time in GC, it's because you're just plain running out of heap. If you're running into GBs of heap, it's almost definitely keeping references to data you don't want it to. The super-cheap-n-easy way to find out what is glomming all of that heap, try passing -XX:-PrintClassHistogram to the JVM and do a stack dump once the heap starts getting full. This will print out how many of each object is being retained and how much heap it consumes.
Thanks folks. There's a lot of helpful stuff in there. I'm bug-hunting at the mo., but when I've nailed the little critter I'll look into this.
ReplyDeleteWe do have a nice thread-sleep time to do a GC - when the machine pauses between layers to allow the previous one to cool (though the impatient young hot-head eD has that time set to 0 on the Bath Darwin...).
So how long are you letting it cool between layers, Adrian?
ReplyDeleteThrowing in another two-cents...
ReplyDeleteWhen I've got an object model in Java that potentially has circular references, I add a clean-up method to the classes. Something like cleanup(). The purpose of this method is to break any references to attribute objects that might in turn have references to this one. The general flow is:
1. assign the attribute to a local variable
2. set the attribute to null
3. if local variable isn't null, call cleanup() on it
This allows you to call cleanup on a container object and have it recurse through the object tree, breaking the references but avoiding an infinite loop.
In the case of the RepRap geometry, you could call cleanup() on a slice after it's been plotted and it would recurse through the lines, points, etc.
After the top-level cleanup() returns, the garbage collector should be able to handle the rest.
If you want to get fancy, you can have all the classes in your object model implement some interface (e.g. Cleanupable) but it's not strictly necessary unless you've got collections of objects where some might have the method and some might not.