[Scons-dev] SCons and Java

[Dirk Bächle]

William,

thanks for your quick reply.

On 31.05.2014 01:39, William Blevins wrote:
> Dirk,
>>
>>     Team,
>>     I have a couple points I'd like to discuss, but for the sake of
>>     organization, I intend to split them into separate emails.
>>     Java Part #2
>>     There has been some discussion on making Scons more ANT-like,
>>     http://www.scons.org/wiki/JavaSupport, and that might solve some
>>     issues like identifying Java targets.  There is merit here
>>     assuming that Java support stays SCons-like at the same time;
>>     otherwise, why not just use ant or maven?
>>
>>         From this I get that you'd like to have some form of Java
>>         support in SCons directly, and don't want to delegate things
>>         to an external program, correct? At least it's my assumption
>>         for the following remarks...
>>
> SCons already has Java support (right?).  I have been a SCons user for 
> a few years and am interested in trying to make the Java toolkit 
> better because I dislike Ant-like build system(s), but I feel that 
> SCons still has room for improvement.  I am braining storming ways to 
> improve the setup.  I have a bunch of idea(s) that I would like to 
> discuss and this just happens to be one of them (#2 in fact).  I know 
> that the SCons teams has talked about refactoring the SCons Java 
> toolkit, but outside the notes from the link above, I haven't seen any 
> further discussions.  I am the new kid on the scons-dev mailing list 
> block, so feel free to give me a tour.  I would be willing to work on 
> SCons with some direction.
That's good to hear, any helping hand is welcome. I haven't been 
involved in the latest Java development, so the link list I gave you in 
the mailing thread 
http://four.pairlist.net/pipermail/scons-users/2014-February/002225.html 
is all I got.

>>     Problem(s) this will solve:
>>     1. Determining the source -> target(s) from *.java -> *.class is
>>     painful, possibly java version dependent, and performance
>>     intensive (possibly); thus, we should optimize this out.
>>     2. Java developers are plagued requiring manual cleaning of a jar
>>     far too often with ant-like system(s) and the current scon(s)
>>     methodology.
>>        1. Clean could be required when deleting or moving a file
>>     since the last build output isn't removed otherwise; avoidable in
>>     current SCons depending on how sources are specified (not obvious
>>     and/or intuitive coming from ant-like system(s)).
>>     3. Simplify the SCons Java API to allow building jar file(s)
>>     without having to interact with *.class file(s) at all.
>>     4. Simplify the SCons Java support for resource file(s);
>>     currently, you have to do copying by hand (if it's supported I
>>     know not where or how).
>>     5. Reduce the amount of code required to build a jar (by a lot).
>>
>>         This sounds like an even more simplified Java Builder than
>>         what we have now. If you could come up with a SoSimpleJava()
>>         Builder, we'd probably add it to the core and let people have
>>         their own choice. 
>>
> That's fair, but since I am talking about not tracking *.class file(s) 
> at all.  The old function(s) couldn't exist.
>>
>>      How I imagine this should work:
>>
>>     1. Treat class file(s) like side-effect(s) and do not include
>>     them (at all) in the dependency tree.
>>         1. This means we don't have to scan the Java file(s) and
>>     determine *.class targets.
>>
>>         I don't think that you can have both. If you treat files as
>>         SideEffects (in the SCons sense), they have to be part of the
>>         dependency DAG. See next comment...
>>
> I meant this in the truer form of side-effect and not the 
> SCons.SideEffects sense.  Getting from A -> C by passing through B, 
> but B is just along the way.  The only way the *.class file(s) could 
> be part of the DAG is if they were identified after the Java compiler 
> had already created the *.class file(s) by basically Glob(*.class).
>
> I would really like to get away from SCons being required to scan Java 
> source file(s) to determine the targets due to the 1 -> N nature of 
> *.java to *.class.  This would resolve several issues with the SCons 
> Java toolkit including incorrect dependency tree(s) with derived 
> (generated) Java source file(s) and some build issue(s) that require 
> manual clean operation(s) on Java code (which may be alleviated 
> depending on how you setup your SConscript, but may not be obvious 
> coming from an ant-like system).
>
Testing out this idea looks like a straightforward task to me. I'd 
really like to encourage you to just go ahead and implement a new 
Tool/Builder in parallel to the existing ones. Then it's easier to 
discuss dis-/advantages and point out the problems of each approach.

However, there certainly will be more obstacles ahead that I can't think 
of right now. It's a perfectly understandable reaction to restrict 
functionality, such that certain issues don't even emerge in first 
place. But one has to be careful about not driving this too far, because 
then it's actually more "problem avoidance" than "problem solving".
In fact, you aren't really dealing with issue number 2.) from your own 
list of "problems this will solve" above. One could argue that you're 
even making it worse, by automating the "make clean; make all" step. So, 
it's *guaranteed* that SCons will rebuild the whole JAR on each single 
source file change...and the user can't do anything to prevent this.

SCons' way of handling builds (in C/C++ for example) is usually to 
prevent the rebuild of intermediate results as much as possible. So be 
aware that you're somewhat swimming against the stream with your idea... ;)

> [...]
>>
>>         You may be able to work around the "clean" problem, by
>>         creating all "*.class" files in a separate folder structure
>>         (pretty much like a VariantDir). But what you'll end up with
>>         is a somewhat crippled DAG, forcing you to remove this output
>>         dir completely and rebuild everything from scratch...even
>>         when only a comment in a single Java source is changed. Only
>>         to "be sure", just like in make or Ant...there is not much of
>>         a win here for the user, I'm afraid.
>>
> The win here is that Java compiler is much faster than the SCons 
> source emitter (I think that's the right term) for Java. There is a 
> performance bottleneck(s) in the Java toolkit which I will discuss in 
> another email; I hope the team is alright with email attachments 
> because I have performance logs.

I'd really like to have a look at your performance measurements. Perhaps 
you can push them up to a public repo, like I did for 
https://bitbucket.org/dirkbaechle/scons_testresults ? Then you can 
publish your full set of data (plus supporting scripts and whatever else 
would be required to reproduce your results), and just have to pass the 
URL around.

>>         Users might also accidentally delete "*.class" files in your
>>         output dir, between two builds. And they expect SCons to
>>         rebuild those intermediate targets on its next invocation.
>>         How do you want to handle this?
>>
> SCons wouldn't rebuild since no source had changed.  This would be an 
> issue if the user was trying to share *.class file(s) which would only 
> be relevant with something like SCons.JavaH, but it supports the 
> directory parameter(s). SCons could check that the directory contents 
> had changed and mark as requiring rebuild; I am grabbing at straws on 
> this one...
>>
>>         Finally, you should take into account that users might want
>>         to create different versions of the same source code. Just as
>>         an example, let's assume they want to create an "original"
>>         and "obfuscated" target(s), or a "simple" and "extended"
>>         release where additional packages get included based on
>>         environment variables. And for maximum speedup they do this
>>         in parallel, by using the "-j" option...so a single dedicated
>>         output dir won't do probably.
>>
> There would be 1 target directory per Java compiler invocation which I 
> imagined would be 1 target directory per *.jar created.  The gain from 
> threading packages in a *.jar file shouldn't be observable until well 
> into 1000+ *.class file(s) per *.jar file.  At this point, I imagine 
> there are bigger architecture concerns at hand than the compile time 
> of 1 *.jar file.
>
It could also be several JARs, that the user tries to build in parallel. 
So you'll have to ensure that the single output dirs don't ever clash, 
by using temporary names for example.

Regards,

Dirk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20140531/49dacedb/attachment-0001.html>