Monday, February 22, 2010
Broadening RepRap Participation
As most of you know, RepRap's software and hardware files are currently hosted on Sourceforge using Subversion. This works fine from a technology perspective, but is not ideal from the point of view of those members of our wider community who want to contribute. In particular, it is difficult to branch the repository in a way that allows core developers only to have write-access to the 'official' stuff, but allows anyone to create branches that can subsequently easily be re-merged.
So, after considering lots of alternatives such as Git, we are thinking (and only thinking at the moment) of moving to Launchpad using Bazaar. This would allow anyone to create branches, to post patches, and so on in a way that could be merged with the official read-only trunk.
There are three equal parts of a RepRap: mechanics, electronics and software. It is particularly important that any system we choose should be as easy to use as possible for electronics and mechanics contributors. This is not to say we value software any the less - on the contrary. It's just that software people will cope well with whatever we choose. So bear that point in mind if you have views.
Speaking of views, before we do anything in this direction, we want yours. Do hit the comment link:
So, after considering lots of alternatives such as Git, we are thinking (and only thinking at the moment) of moving to Launchpad using Bazaar. This would allow anyone to create branches, to post patches, and so on in a way that could be merged with the official read-only trunk.
There are three equal parts of a RepRap: mechanics, electronics and software. It is particularly important that any system we choose should be as easy to use as possible for electronics and mechanics contributors. This is not to say we value software any the less - on the contrary. It's just that software people will cope well with whatever we choose. So bear that point in mind if you have views.
Speaking of views, before we do anything in this direction, we want yours. Do hit the comment link:
Comments:
<< Home
Git is my tool of choice, my recommendation, and my vote.
But, whether you move to Bazaar, Mercurial, or Git, moving to a decentralized versioning system for this project is commendable. I think it's far more in line with the Open Source ideals inherent to the RepRap project.
So, in short, I approve.
But, whether you move to Bazaar, Mercurial, or Git, moving to a decentralized versioning system for this project is commendable. I think it's far more in line with the Open Source ideals inherent to the RepRap project.
So, in short, I approve.
Having looked at the alternatives. I would probably prefer git over the others if only for its distributed capabilities and it can branch and merge too (as well as rebase).
Something that supports a tree structure and as sugested a sensible degree of seperation I think would be good too.
Something that supports a tree structure and as sugested a sensible degree of seperation I think would be good too.
Nop. If the electronics files are text and(kicad as an example) are treated as source it should work well enough to get away with. Binary file formats may present more of a problem though.
Don't know if git would handle this without trying it.
Don't know if git would handle this without trying it.
If you ask me, it doesn't really matter (that much) which version control system you will use. Of course, some may be suited better or be easier to use.
But the more important questions are:
* How to represent the mechanics and the electronics in a way so that changes can be merged?
* The files you use for designing the system should be readable/editable by open source or at least free/inexpensive software. Otherwise people will work with other software making reintegration very hard
* There should be a well defined process to reintegrate improvements into the main branch. I think it is not very helpful if everybody creates their own version.
just my 2c
But the more important questions are:
* How to represent the mechanics and the electronics in a way so that changes can be merged?
* The files you use for designing the system should be readable/editable by open source or at least free/inexpensive software. Otherwise people will work with other software making reintegration very hard
* There should be a well defined process to reintegrate improvements into the main branch. I think it is not very helpful if everybody creates their own version.
just my 2c
My votes would go for git in combination with github.com. It is super easy to fork the project, change or modify some code and do a pull request. The maintainer(s) of the original repository can than decide to commit the 'patch' to the original repository. This 'workflow' really does increase participation.
I recently switched to git for most of my needs (from svn). Since bazaar offers support for using git as a client (via bzr git-serve) I think bazaar will do just fine.
In my new job we are using bzr and launchpad. you cannot beat the benefits of launchpad for a distributed code development process. Its not just git or bzr that you should look at, but the whole process. (disclaimer: i work for canonical, but this is my opinion, not marketing)
Git has a far larger community base and is appearing more and more to be the choice of many projects. Technical differences aside, few really, what it takes for newbies to get up and running should be the main point of choice.
BZR while very capable has a much different flavour of support, assuming quite a bit of knowledge.
BZR while very capable has a much different flavour of support, assuming quite a bit of knowledge.
Re: branching and merging electronics and mechanical components. In the SKDB project, they use a plain-text file to represent relationships between components (mechanical, no electronics demo yet though). When you make changes to these files, `diff` and other tools are more easily able to handle those changes. CAD data is another story. I suggest bringing up the topic here and searching the archives there too.
Git is good. There are many tutorials and many people to ask for help with it in most areas of design of code or hardware.
Actually, come to think of it, if you want to be able to do merging between CAD files, the best way to do this right now would be something akin to (1) BRLCAD mged scripts and merging text files of those scripts, (2) OpenSCAD text files (before they are compiled into STL) which can be merged just like in code, and (3) some sort of higher-level API that pythonOCC has been working on like in Level2API.py. For instance, consider some python used to generate CSG (hypothetical only). This would allow merges and changes with 'diff' and other dvcs tools like git, hg, bzr, etc.
That's a simple quick hack to things. There's a deeper problem that will bite you in the ass re: automatic merging. At the moment, tools like 'diff' don't know how to merge different CAD features. For instance, one user might have added handle bars on to a bike design, and another user does the same, and then a third user tries to merge the designs. That will not go over well and still requires user intervention. Buut in the mean time, text-based CAD file specification gives us a lot of power. :-) Cheers!
- Bryan
(btw, meet up with me in #hplusroadmap on freenode)
That's a simple quick hack to things. There's a deeper problem that will bite you in the ass re: automatic merging. At the moment, tools like 'diff' don't know how to merge different CAD features. For instance, one user might have added handle bars on to a bike design, and another user does the same, and then a third user tries to merge the designs. That will not go over well and still requires user intervention. Buut in the mean time, text-based CAD file specification gives us a lot of power. :-) Cheers!
- Bryan
(btw, meet up with me in #hplusroadmap on freenode)
I've been happy with hosting Gladex on Launchpad for the past few years. I really like the bug tracking and Blueprint systems. The blueprints allow users to request features, which could be a really good way for people to express their ideas to the community in a collected way and allow those ideas to be accepted as specifications for the project.
Launchpad also has systems for translations, package building and hosting, answer tracking and FAQs, and mailing lists.It just such a great place to organize things.
I'm not informed enough to debate git vs. bzr, but I do feel strongly that Launchpad is a great way to organize a project.
I haven't tried it yet, but here appears to be "a plugin for bzr to work with git trees."[1] Using git may not preclude the use of Launchpad.
[1] https://launchpad.net/bzr-git
Launchpad also has systems for translations, package building and hosting, answer tracking and FAQs, and mailing lists.It just such a great place to organize things.
I'm not informed enough to debate git vs. bzr, but I do feel strongly that Launchpad is a great way to organize a project.
I haven't tried it yet, but here appears to be "a plugin for bzr to work with git trees."[1] Using git may not preclude the use of Launchpad.
[1] https://launchpad.net/bzr-git
I think there's some important things being missed; most importantly, the processes are far more important then the software. Are you committed to actually opening up development, and pulling in patches from the community?
This project has been through two wikis, neither of which have been terribly welcoming. Are you committed to changing that?
This project has been through two wikis, neither of which have been terribly welcoming. Are you committed to changing that?
While I haven't used git directly, I used BitKeeper, which git was based off of, and it was excellent. I've heard great things about git. Until today I hadn't heard anything about bazaar.
I also agree with the statement that it's very important to figure out how to represent the project (directory structure wise, etc). If someone finds that a Mendel part wasn't rotated correctly, rotates it, and checks it in, that's fine - that could easily be merged into trunk without objections.. If someone however modifies the frame vertex piece to have a little nub on the side to hold string because they were making some modification to allow you to make your Mendel into a tent, well, that's different, and there's value in holding that somewhere for public consumption while not officially moving it into the official project.
If solid files and electronics files are opaque object files that don't benefit much from diff/merge tools, then I'd argue that the software portion actually matters more (with respect to source code tools), since you want to be able to look at changes over time, debug problems, etc. If the merge tools only work well for source code, have source code be the deciding factor (providing that the source code control system can handle binary files, which at least bitkeeper was perfectly able to do).
I also agree with the statement that it's very important to figure out how to represent the project (directory structure wise, etc). If someone finds that a Mendel part wasn't rotated correctly, rotates it, and checks it in, that's fine - that could easily be merged into trunk without objections.. If someone however modifies the frame vertex piece to have a little nub on the side to hold string because they were making some modification to allow you to make your Mendel into a tent, well, that's different, and there's value in holding that somewhere for public consumption while not officially moving it into the official project.
If solid files and electronics files are opaque object files that don't benefit much from diff/merge tools, then I'd argue that the software portion actually matters more (with respect to source code tools), since you want to be able to look at changes over time, debug problems, etc. If the merge tools only work well for source code, have source code be the deciding factor (providing that the source code control system can handle binary files, which at least bitkeeper was perfectly able to do).
Just a thought, It may help to look at the project as being a program of projects (or Project of projects) instead of a monolithic project with areas.
Yup it appears trivial to say this but it sometimes can radicaly affect the way you might tackle organizing it.
Yup it appears trivial to say this but it sometimes can radicaly affect the way you might tackle organizing it.
A few thoughts:
* As so many others have said: process over tools!
* Please treat stable releases as a priority: software, documentation on how to build the mechanical structure, where to source the parts, ...
* As so many others have said: process over tools!
* Please treat stable releases as a priority: software, documentation on how to build the mechanical structure, where to source the parts, ...
However we do the code changes, there are a lot of people like myself that are used to a CAD style of mechanical design, for whom tools like AOI and OpenScad are difficult to use.
It's worth noting that both the Darwin and the Mendel assemblies were designed with Solid Edge (at least the versions I've found in Subversion), CAD software that is very good, but essentially out of reach for anyone not in a UK university due to its cost. As well, since they were designed with the educational version of Solid Edge, nothing but the educational version of Solid Edge can open the original designs. The STEP files do import into other CAD packages quite nicely though.
It would be nice if there were a way for us to include changes in the form of STL or better yet STEP files to mechanical objects. This would probably have to be done on a part by part basis, as opposed to a feature by feature basis.
I don't think using AOI or OpenSCAD as the file format for everything is a good idea yet - those tools are not developed enough at this point for everyone to use them reasonably.
I think that whatever system we use to keep track of CAD designs, it should be format agnostic. STL, although a terrible file format in many respects, does have the advantage of being the lowest common denominator. STEP is more accurate, but not many of the open source 3D editors support it yet.
It's worth noting that both the Darwin and the Mendel assemblies were designed with Solid Edge (at least the versions I've found in Subversion), CAD software that is very good, but essentially out of reach for anyone not in a UK university due to its cost. As well, since they were designed with the educational version of Solid Edge, nothing but the educational version of Solid Edge can open the original designs. The STEP files do import into other CAD packages quite nicely though.
It would be nice if there were a way for us to include changes in the form of STL or better yet STEP files to mechanical objects. This would probably have to be done on a part by part basis, as opposed to a feature by feature basis.
I don't think using AOI or OpenSCAD as the file format for everything is a good idea yet - those tools are not developed enough at this point for everyone to use them reasonably.
I think that whatever system we use to keep track of CAD designs, it should be format agnostic. STL, although a terrible file format in many respects, does have the advantage of being the lowest common denominator. STEP is more accurate, but not many of the open source 3D editors support it yet.
I myself am partial to git and github.com. Github really makes it easy to fork code. However, I wouldn't mind a bzr and Launchpad combination either.
For example in the mobile communication field where both Midge and I were employed until recently.
Project requirements, Hardware Architecture, Software Architecture, UI design, Hardware design, Software design, functional specifications, test specifications and test results were all controlled by the same document management system for each project.
The whole process would need some configuration management to allow for Branching and merging of any types of documents in and out of the project main streams.
From what I have seen of Source forge or Git both seem to offer the facilities to provide the functionality required to do this.
The more difficult part is defining the procedure that everyone has to follow then providing quality management of the final system.
Project requirements, Hardware Architecture, Software Architecture, UI design, Hardware design, Software design, functional specifications, test specifications and test results were all controlled by the same document management system for each project.
The whole process would need some configuration management to allow for Branching and merging of any types of documents in and out of the project main streams.
From what I have seen of Source forge or Git both seem to offer the facilities to provide the functionality required to do this.
The more difficult part is defining the procedure that everyone has to follow then providing quality management of the final system.
I use git via Sourceforge for ShareRap and think its a great tool. I have been a Sourceforge user/project contributor since 2002. I'm not sure there is a compelling reason to leave Sourceforge for launchpad.
I think there's some important things being missed; most importantly, the processes are far more important then the software. Are you committed to actually opening up development, and pulling in patches from the community?
Yes. This post/RFC for the repository is part of it, but there is a lot of stuff changing behind the scenes.
This project has been through two wikis, neither of which have been terribly welcoming. Are you committed to changing that?
Everything is getting moved over into a single wiki.
And more generally, Yes!. I plan to introduce social networking tools to the wiki, add a "New Project" button for people who run into trouble with mediawiki code, and other features, like rendering of uploaded files.
Also, we'll be starting a
RepLab - RepRap Fab Lab
which covers much of the same ground the Open Manufacturing group is interested in. Basically, designing and documenting tools, many of them made with RepRaps, many of them capable of making RepRaps but welcome to every fabrication method there is. For that, we'll need to be sure to have a positive culture and welcoming server.
I need to email the skdb group and see if they want to help us on this. (By merging, maybe.) If we don't coordinate, they're going to be busy trying to be a superset of us, while we're busy being a superset of everything.
Regarding CAD and file formats, we're becoming very file format agnostic. Because if we mandate "Use Program X", all the W-users, Y-users, and Z-users just evaporate. Mind you, people who maintain one system or set of systems are curiously ambivalent about integrating or even tracking non-X-format contributions, but that's an unsolvable problem. Or, at least, I know it's not my problem.
Arguing about tools can waste a lot of time. Best is just to be polite about it, make sure the server and community are maximally welcoming, and have fun RepRapping.
Yes. This post/RFC for the repository is part of it, but there is a lot of stuff changing behind the scenes.
This project has been through two wikis, neither of which have been terribly welcoming. Are you committed to changing that?
Everything is getting moved over into a single wiki.
And more generally, Yes!. I plan to introduce social networking tools to the wiki, add a "New Project" button for people who run into trouble with mediawiki code, and other features, like rendering of uploaded files.
Also, we'll be starting a
RepLab - RepRap Fab Lab
which covers much of the same ground the Open Manufacturing group is interested in. Basically, designing and documenting tools, many of them made with RepRaps, many of them capable of making RepRaps but welcome to every fabrication method there is. For that, we'll need to be sure to have a positive culture and welcoming server.
I need to email the skdb group and see if they want to help us on this. (By merging, maybe.) If we don't coordinate, they're going to be busy trying to be a superset of us, while we're busy being a superset of everything.
Regarding CAD and file formats, we're becoming very file format agnostic. Because if we mandate "Use Program X", all the W-users, Y-users, and Z-users just evaporate. Mind you, people who maintain one system or set of systems are curiously ambivalent about integrating or even tracking non-X-format contributions, but that's an unsolvable problem. Or, at least, I know it's not my problem.
Arguing about tools can waste a lot of time. Best is just to be polite about it, make sure the server and community are maximally welcoming, and have fun RepRapping.
Also, to continue the derail from ?git/buzz/etc?, please note that we have very active forums-mailing lists
http://dev.forums.reprap.org/index.php.
I think we'll have a new Polymer Working Group in a few hours, for example. Mailing list functionality for all forums will take a few more days.
http://dev.forums.reprap.org/index.php.
I think we'll have a new Polymer Working Group in a few hours, for example. Mailing list functionality for all forums will take a few more days.
Why not use wikidot. It is open source, and you can change file permissions, etc. It is fairly easy to learn and use. In fact, the makerbot wiki is hosted on wikidot.
See:
http://makerbot.wikidot.com/
Unfortianately, Reprap.wikidot.com is already taken, but that's alright. It's just an idea.
See:
http://makerbot.wikidot.com/
Unfortianately, Reprap.wikidot.com is already taken, but that's alright. It's just an idea.
Why not use wikidot.
Since we're nearly done with mediawiki transition, I'm utterly reluctant to play musical chairs to a new wiki.
Plus, since our server hosts our mediawiki, we can do strange but possibly valuable things to it, like work with skdb and learn how to use opencascade as an stl (or STEP) renderer.
If we run into a wall with mediawiki, we'll extend it or replace it.
----------------------------
Meanwhile, git, bzr? What do people like?
Also, does it make sense to use an exernal bug tracker like RT: Request Tracker or do we use the built in repository tool?
How well do the built in repository tools track things like "I built a RepStrap using: x-positioning bot, y-boards, z-controller software, w-extruder and I get good results extending RepRap research, but I have this bug _____"
Because we're using a host of tools like ReplicatorG or EMC, or important but off-site electronics which may never get into the new repository, but we want to track bugs against these important tools.
Since we're nearly done with mediawiki transition, I'm utterly reluctant to play musical chairs to a new wiki.
Plus, since our server hosts our mediawiki, we can do strange but possibly valuable things to it, like work with skdb and learn how to use opencascade as an stl (or STEP) renderer.
If we run into a wall with mediawiki, we'll extend it or replace it.
----------------------------
Meanwhile, git, bzr? What do people like?
Also, does it make sense to use an exernal bug tracker like RT: Request Tracker or do we use the built in repository tool?
How well do the built in repository tools track things like "I built a RepStrap using: x-positioning bot, y-boards, z-controller software, w-extruder and I get good results extending RepRap research, but I have this bug _____"
Because we're using a host of tools like ReplicatorG or EMC, or important but off-site electronics which may never get into the new repository, but we want to track bugs against these important tools.
As a software developer, I've used both git and bazaar on large software projects professionally (in my day job).
Both git and bazaap will do a good job and support a very similar feature set (branching, distributed development, etc - although the terminology is different). However, if I had to choose, I've found git easier to learn and prefer git to bazaar. Git is slightly faster, and seems to be better at picking up changes to files (finding moved files, copying/splitting code into other files) and does a better job of merging changes.
Both handle binary files (like board /hardware files, anything non-text) well and both have plugins for eclipse, windows clients, and good open-source support. 'getting started' documentation is easier to find on git, IMHO.
Moving to a new library, I wonder if it is worth splitting the current 'big' library into two or three sections - the core Java/host software, a firmware section, and a hardware repository? It's currently 170Meg+ - not a problem yet, but a pretty large download.
Both git and bazaap will do a good job and support a very similar feature set (branching, distributed development, etc - although the terminology is different). However, if I had to choose, I've found git easier to learn and prefer git to bazaar. Git is slightly faster, and seems to be better at picking up changes to files (finding moved files, copying/splitting code into other files) and does a better job of merging changes.
Both handle binary files (like board /hardware files, anything non-text) well and both have plugins for eclipse, windows clients, and good open-source support. 'getting started' documentation is easier to find on git, IMHO.
Moving to a new library, I wonder if it is worth splitting the current 'big' library into two or three sections - the core Java/host software, a firmware section, and a hardware repository? It's currently 170Meg+ - not a problem yet, but a pretty large download.
As for bug tracking, I've used several systems but nothing I'd particularly recommend over anything else. You probably want a dedicated separate system, although the highly customised nature of the reprap development may make identifying common bugs hard, given the mix of steppers/drivers/firmware/boards/software environments out there. Restrict your bug reporting to the core software and rely on forums/wikis/chat for other support - provision of a place to report 'bugs' also brings an expectation that someone else might fix them for you. :-) It's almost impossible to diagnose a hardware /software issue without the ability to reproduce the problem accurately, and repraps are heavily hardware biased.
:-)
:-)
Restrict your bug reporting to the core software and rely on forums/wikis/chat for other support - provision of a place to report 'bugs' also brings an expectation that someone else might fix them for you. :-) It's almost impossible to diagnose a hardware /software issue without the ability to reproduce the problem accurately, and repraps are heavily hardware biased.
That would simplify things, but it would start to have a corrosive effect on important projects that are not blessed as being "core", like the Laser Cutter folk may develop, etc.
It would also tend to lock out projects that start as non core and become of greater and greater importance. I'd prefer to treat non-core RepRap user-developers with much more respect than that, and I expect a higher degree of commitment from them - they should take responsibility for their bugs if they're willing to check their work into our repository.
That would simplify things, but it would start to have a corrosive effect on important projects that are not blessed as being "core", like the Laser Cutter folk may develop, etc.
It would also tend to lock out projects that start as non core and become of greater and greater importance. I'd prefer to treat non-core RepRap user-developers with much more respect than that, and I expect a higher degree of commitment from them - they should take responsibility for their bugs if they're willing to check their work into our repository.
This is something that I feel I can give valuable feedback to. I am chieftain of the widelands project (http://wl.widelands.org) and we recently moved to launchpad after I used git, hg and bzr in many smaller private projects.
Widelands is a big project. We have many, many binary files of various sizes (also big ones ~20MB) in our repo and the choice of launchpad was one of integration before tools. I personally prefer git because it is much more powerful; but especially for the developers coming from SVN, bzr is easier to handle since it is similar (git revert <-> bzr revert <-> svn revert as an example).
But launchpad has hands down the best community integration of them all (including github): bug reporting is straight forward and easy, branching/merging is easy, web based translations are king and bzr<->launchpad integration is killer. Bzr/launchpad has been handling our repo with ease and stacked branches makes pushing/pulling new branches dirt cheap.
My 2 c. Other tools are great too, but launchpad + bzr is a solid decision for big process that want to makes things easy for drive by contributors.
Cheers,
Holger
Widelands is a big project. We have many, many binary files of various sizes (also big ones ~20MB) in our repo and the choice of launchpad was one of integration before tools. I personally prefer git because it is much more powerful; but especially for the developers coming from SVN, bzr is easier to handle since it is similar (git revert <-> bzr revert <-> svn revert as an example).
But launchpad has hands down the best community integration of them all (including github): bug reporting is straight forward and easy, branching/merging is easy, web based translations are king and bzr<->launchpad integration is killer. Bzr/launchpad has been handling our repo with ease and stacked branches makes pushing/pulling new branches dirt cheap.
My 2 c. Other tools are great too, but launchpad + bzr is a solid decision for big process that want to makes things easy for drive by contributors.
Cheers,
Holger
I'm not sure why, but I find launchpad deeply frustrating to use -- I'm a greybeard Debian Developer though, so maybe I'm just too set in my ways.
Anyway, I'd also favour git, and suggest that you look at ikiwiki as a way of doing your bug tracking in git.
Note, I'm not saying that you need to host your whole wiki on ikiwiki, just that using that as a tool to expose bugs hosted in the same git repository gives people the ability to do things like fix a bug and update the bug report in the same commit, allowing for things like doing that off-line against a local repository when in their shed, and then pushing it back later.
See Ikiwiki's own Bugs page as an example.
Anyway, I'd also favour git, and suggest that you look at ikiwiki as a way of doing your bug tracking in git.
Note, I'm not saying that you need to host your whole wiki on ikiwiki, just that using that as a tool to expose bugs hosted in the same git repository gives people the ability to do things like fix a bug and update the bug report in the same commit, allowing for things like doing that off-line against a local repository when in their shed, and then pushing it back later.
See Ikiwiki's own Bugs page as an example.
Mind you, I haven't tried git or bazaar, but I like the other parts of launchpad, though I have a few minor issues with the bugtracker, it's too minimalistic, but then again my tracker of choice is Mantis.
Post a Comment
<< Home