Log for #openttdcoop.devzone on 31st May 2014:
Times are UTC Toggle Colours
06:11:36  *** LSky` has joined #openttdcoop.devzone
07:40:29  *** Alberth has joined #openttdcoop.devzone
08:18:33  *** LSky` has quit IRC
08:31:32  *** gelignite has joined #openttdcoop.devzone
08:57:33  <planetmaker> Alberth, nfo output is broken with nmlc:
08:58:32  <Alberth> nice :)
08:59:08  <planetmaker> also bytearray stuff, I assume...
08:59:20  <Alberth> how did that get through the regression? Or doesn't it check nfo?
08:59:44  *** andythenorth has joined #openttdcoop.devzone
08:59:48  <Alberth> but nfo is text, shouldn't be bytes
08:59:52  <Alberth> hi andy
09:00:17  <planetmaker> regression should check also nfo
09:00:37  <andythenorth> lo
09:02:57  <Alberth> check is clearly incomplete :p
09:03:06  <planetmaker> yup :)
09:04:57  <Alberth> self.file.write('\xC3\x9E'.decode('utf-8'))    yep, that looks not good :p
09:05:34  <planetmaker> <-- so it is tested. But... yes... insufficiently :)
09:06:04  <Alberth> just add a unicode character in some string?
09:06:17  <Alberth> ie how to reproduce your issue?
09:08:01  <planetmaker> nmlc --nfo firs.nfo firs.nml
09:10:11  <andythenorth> educate me a bit
09:10:23  <andythenorth> currently an ID is not explicitly scoped to local or global
09:10:46  <andythenorth> it’s implicitly global unless it gets redefined lower down the code
09:10:59  <andythenorth> and then it becomes local
09:11:44  <andythenorth> sound right?
09:16:04  <Alberth> afaik it's still the same ID, but it points to a new thing
09:16:37  <Alberth> a = 5;   a = 6;    <- same "a", but with different content
09:24:10  <Alberth> planetmaker:
09:25:05  <andythenorth> and our belief is that there’s a slow step during compile, where nmlc has to walk the code, looking for ids, then figuring out what they resolve to in the resulting newgrf?
09:25:21  <andythenorth> I’m not sure what they actually resolve to in the newgrf, but I’m assuming they’re pointers
09:25:21  <Alberth> last line does get re-added in the raw display :p
09:25:39  <planetmaker> I bust added the b in the first hunk, yes. The rest is additional regression test, yes?
09:26:12  <Alberth> yes, added a  ß   in some text
09:26:40  <Alberth> andythenorth: I don't know what part in nml is actually slow
09:27:03  <Alberth> the graphics were slow, but they are cached now
09:27:24  <andythenorth> I think it’s been assumed that the resolving IDs part is slow
09:27:29  <planetmaker> what's the reason to write the thorn char directly (instead of byte encoding it)?
09:27:38  <andythenorth> I just wonder if it’s true, before we go introducing a linker :P
09:28:31  <planetmaker> anyway, I don't mind either way, especially as it's still there as comment :)
09:28:34  <Alberth> planetmaker: we have a language capable of unicode source code, so why decode it manually?
09:28:36  <planetmaker> works for me, thus +1
09:28:51  <planetmaker> fair enough, indeed
09:29:43  <planetmaker> Teßt Stätiöµſ
09:29:45  <Alberth> andythenorth: the idea of a linker is that you split up your grf in many small more-or-less independent parts, so you don't need to rebuild everything on each change
09:30:28  <Alberth> planetmaker: you want that text instead?
09:31:37  <andythenorth> Alberth: I know :) I just wonder if it solves the right problem
09:31:43  * Alberth assumes you do
09:31:46  <andythenorth> maybe that’s moot, maybe a linker is just a sane step
09:32:04  <andythenorth> I just wonder if there are simpler ways to go faster
09:33:18  <planetmaker> Alberth, no, the thorn is quite fine
09:33:21  <Alberth> interesting english test file now :p
09:33:40  <planetmaker> :P
09:34:17  <planetmaker> I just made a stab at it, so my quick fix differed from yours. Yours makes more sense
09:35:51  <Alberth> I was talking about Teßt Stätiöµſ   :)
09:36:01  <Alberth> pushed btw
09:36:29  * andythenorth wishes he was smarter
09:36:53  <andythenorth> trying to figure out how IDs end up in a symbol table, without collisions, etc
09:37:18  <andythenorth> in manually coded nfo, it’s fricking trivial, you just manage IDs yourself and try not to screw up
09:37:23  <Alberth> I am not even sure they get re-used ever
09:37:29  <planetmaker> oh :)
09:37:40  <planetmaker> well, might be that string is never re-used. Dunno
09:38:05  <planetmaker> Alberth, IDs do get re-used. Or nml would simply not work. There's only 255
09:38:12  <andythenorth> that is the tricky bit
09:38:26  <andythenorth> because there’s zero indirection in NFO, you just manage them yourself when writing code
09:38:40  <andythenorth> the indirection in nml is nice for authors, but makes the problem harder
09:38:44  <DevZone> Project opengfx-mars build #127-push: SUCCESS in 2 min 24 sec:
09:38:48  <Alberth> finding the last ue is not that complicated
09:38:51  <Alberth> *use
09:38:52  <DevZone> Project Townnames - German build #86-push: SUCCESS in 8.7 sec:
09:38:53  <DevZone> Project NML - NewGRF Meta Language build #349-push: SUCCESS in 3 min 49 sec:
09:39:24  <Alberth> just walk through the uses of identifiers from the back
09:40:07  <planetmaker> nml uses a push / pop stack of used IDs to get the next free one
09:40:18  <andythenorth> and would introducing explicitly declared globals simplify anything?
09:40:25  <planetmaker> iirc
09:41:02  <Alberth> andythenorth: if you want to know what the slow part of nml, do a profiled run of nml, and see whether you can pinpoint the dominant function
09:41:36  <andythenorth> I think yexo did that
09:41:42  <andythenorth> I don’t know where the results are though
09:41:55  <andythenorth> I have to learn cProfile?
09:42:47  <planetmaker> yeah. and then teach me :)
09:42:55  <andythenorth>
09:42:57  <Webster> Title: PythonSpeed/PerformanceTips - Python Wiki (at
09:44:25  <andythenorth> also
09:44:26  <Webster> Title: 26.4. The Python Profilers Python v2.7.7rc1 documentation (at
09:45:05  <Alberth> andythenorth: wrong python version
09:45:14  <andythenorth> oh :)
09:45:17  <andythenorth> yes I forget
09:45:24  <Alberth> if you use trunk nml, at least
09:45:31  <andythenorth> I’ll have to change my nml setup soon :P
09:45:47  <Alberth> soon(tm)   :)
09:45:52  <andythenorth> it’s borked anyway right now, I seem to have 2.7 nml with a 2.6 PIL
09:45:58  <andythenorth> it shouldn’t work, but it does :P
09:46:14  <andythenorth> but throws some whining errors back from PIL
09:53:57  <andythenorth> so if we had a linker
09:54:15  <andythenorth> what would the interface be?  It automagically compiles only changed files?
09:54:54  <Alberth> basic idea is the same as with c/c++
09:56:08  <Alberth> for i in a b c d ; do nmlc_partly  $i.nml  -o $i.linkinput; done
09:56:08  <Alberth> nmlc_link *.linkinput  foo.grf
09:58:38  <andythenorth>  ‘a b c d’ are?
09:58:45  <andythenorth> list of files?
09:58:50  <andythenorth> objects?
09:58:53  <Alberth> a.nml b.nml c.nml d.nml
09:59:07  <Alberth> part of file names
09:59:25  <Alberth> it does need declaration of symbols though
09:59:54  <Alberth> eg your sprite definition may be in one file, and you want to use it in another file
10:00:08  <andythenorth> that’s what I can’t grok
10:00:23  <andythenorth> I can’t see how automagic symbols would work
10:00:27  <andythenorth> I don’t even like them tbh
10:01:46  <Alberth> file1:   x = spritedefinition(.....);
10:01:46  <Alberth> file2:  extern spritedefinition x;      make_industry(... x ... );
10:03:13  <Alberth> file1.linkinput then says "I have x, which is a sprite definition defined at offset ... here"
10:03:13  <Alberth> file2.linkinput says "I need a reference to x, which should be a sprite definition"
10:03:59  <andythenorth> does a symbol table manage the references?
10:04:02  <Alberth> the linker then places file1 and file 2 after each other, assigns an id to x in file 1, and fills that same number at the use of in file 2
10:04:44  <Alberth> the linker builds a symbol table, and uses it to decide order of the files, and when a symbol can be discarded
10:04:53  <planetmaker> so the file with the grf { } always goes first as it defines the (global) parameters, too
10:06:07  <andythenorth> so authors need to manage globals?
10:06:08  <Alberth> theoretically, you may never use the parameters, or only just in the last file or so
10:06:47  <planetmaker> well. Also practically. As most actions are not valid prior to the grf definition in a grf file
10:07:08  <planetmaker> and indeed, it may define a bunch of pointless parameters
10:07:42  <Alberth> the linker should probably do something special for grf then
10:07:43  <planetmaker> andythenorth, if you want linking, you need to deal with concept of local and global. I see not much way around that
10:08:09  <andythenorth> I am +1 to it
10:08:30  <planetmaker> action14 (description,...) is valid only *prior* to action8, though. But that's part of our grf block, too
10:08:37  <andythenorth> as you’ll see from my use of numeric IDs for items….I don’t trust the magic
10:08:38  <andythenorth> :)
10:09:19  <planetmaker> well, giving them explicitly is explicitly forseen in the nml specs
10:09:21  <Alberth> you can build dependence trees prior to compilation
10:09:44  <Alberth> which would mean you don't specify anything, the compiler will figure it all out
10:10:00  <Alberth> but that's a layer on top of the basic linker
10:10:28  <andythenorth> I am dubious about the practicality of that
10:10:47  <andythenorth> I think it would end up walking the entire grf, and possibly needing to rewrite IDs
10:11:07  <Alberth> you just keep all source in memory
10:11:15  <Alberth> eg in an eclipse ide or so
10:11:45  <Alberth> or you write the dependencies to disk as a cache
10:12:18  <Alberth> andythenorth: you never write any specific ID until the final link
10:12:38  <Alberth> everything is symbolic through the variable name you give to it
10:12:54  <andythenorth> hmm
10:12:59  <andythenorth> it’s probably simpler than I think
10:13:27  <planetmaker> I have the feeling alberth knows some pieces of compiler theory :)
10:13:28  * andythenorth used to write varact 2 chains with IDs like: BB, BB, BB, BB, BB
10:13:35  <andythenorth> which worked rather well
10:14:06  <Alberth> planetmaker: but I never wrote a linker :)
10:14:16  <planetmaker> :) Ah, the challenge! :)
10:14:31  <Alberth> andythenorth: what does "BB" mean?
10:14:38  <planetmaker> just a numeric id
10:14:39  <andythenorth> ID of the next varact 2
10:14:48  <andythenorth> - next
10:14:52  <andythenorth> ID of the varact 2
10:15:04  <andythenorth> we probably have multiple meanings for ‘ID'
10:15:08  <andythenorth> due to nfo / nml
10:15:40  <Alberth> yeah, and assumptions that I know what varact2 does :p
10:16:00  <planetmaker> well, they mean basically the same. An ID is the name of a switch/(var)action2 or an action0/property
10:16:03  <andythenorth> you might honestly be better without legacy nfo-writing baggage :P
10:16:12  <planetmaker> switch (...) <<-- varaction2
10:16:26  <andythenorth> I have trouble sometimes because I’m projecting nml onto nfo, inappropriately
10:17:03  <Alberth> nfo is eventually the target, so I want to know it too :)
10:17:13  <Alberth> or rather the binary form of nfo :p
10:17:17  <planetmaker> well, rather grf :)
10:17:18  <planetmaker> yeah
10:17:31  <andythenorth> how much does grf differ from nfo?
10:17:41  <Alberth> but that's almost 1-to-1 afaik
10:17:42  <planetmaker> it's binary contrary to text
10:17:42  <andythenorth> I never wanted to poke that question too closely before :P
10:17:54  <planetmaker> though nfo text is a string of hex numbers for most part
10:18:10  <Alberth> yep :
10:18:14  <Alberth> :)
10:18:18  <planetmaker> but it has control sequences like \b \w \d G etc
10:18:31  <Alberth> control sequences are for the weak :p
10:18:40  <planetmaker> also explicit strings "bla bla"
10:18:41  <andythenorth> sugar pills
10:19:16  <Alberth> I never understood why it doesn't have a sane parser, it would make things so much easier to write down
10:19:31  <Alberth> probably hysterical raisins
10:19:43  <planetmaker> and some syntactic sugure like the format to write a spirte -1 * <size> <actionbyte> <...>
10:20:02  <Alberth> yep :)
10:20:07  <planetmaker> *sugar
10:20:18  <andythenorth> Alberth: my nfo commenting appears to induce rage in others…but HEQS is fairly fully commmented nfo
10:20:22  <Alberth> not to mention the obligatory 3 lines of comment :)
10:20:24  <andythenorth> might not be a good place to start
10:20:45  <planetmaker> and grf includes somehow the sprites instead of a file reference
10:21:08  <Alberth> it just pulls out the pixels, I think
10:21:26  <planetmaker> yeah, mostly. I think some RLE thrown on it, though
10:21:39  <planetmaker> optionally
10:22:28  <Alberth> andythenorth: that's quite close to nml, isn't it?
10:22:45  <andythenorth> Alberth: I don’t think an nml code would agree
10:22:51  <andythenorth> if they had no understanding of nfo :)
10:23:07  <andythenorth> coder *
10:23:20  <Alberth> yeah, it misses some english words
10:23:31  <planetmaker> well... some are in the comments :)
10:23:46  <andythenorth> I found it pretty easy to adapt, once I got past the install and some magic
10:24:06  <andythenorth> I don’t find it any easier to write ‘switch_vehicle_length_depot_view’ than ‘E2’ though
10:24:17  <andythenorth> they’re about the same
10:24:31  <andythenorth> except when you hit FF and you have to rethink your IDs :P
10:24:37  <Alberth> I have the problem the other way around, I can see nml, but have little clue how that translates to nfo/grf
10:25:07  <Alberth> ie how much extra work nml is doing except a bit of template expansion
10:25:27  <planetmaker> I think the bonus is in readibility, so you somewhat understand the code without implicit knowledge
10:25:56  <andythenorth> well the fundamental nml stuff mostly maps to action 0 / 1 / 2 / 3
10:26:06  <andythenorth> then there are a lot of nml convenience features
10:26:09  <planetmaker> yes. That's fundamental how grfs work
10:26:25  <andythenorth> there are some important detail differences in how nml hides away parts of the action 2 / 3 chain
10:26:28  <planetmaker> and the bonus of nml is the ease to chain complicated switches
10:26:43  <Alberth> no big translations / rewrites thus to a different structure or so
10:26:53  <planetmaker> you can write one nml switch which translates to several dozens of action2 IDs in nfo
10:27:19  <andythenorth> all of which need IDs managed …. :)
10:27:32  <andythenorth> FIRS hits the ID limits btw
10:27:38  <andythenorth> in spritegroup calcs
10:27:46  <andythenorth> I have to rewrite it some time
10:27:52  <planetmaker> hu?
10:27:54  <Alberth> grfspec problem :p
10:28:07  <planetmaker> I haven't seen that error yet anywhere with FIRS
10:28:15  <planetmaker> Alberth, we could extend those :P
10:28:25  <planetmaker> (maybe)
10:28:55  <andythenorth> planetmaker: it’s in spritelayouts somewhere
10:28:55  <Alberth> no idea if that's needed
10:29:08  <andythenorth> we have something like slope-aware, fence-aware, snow-aware, date aware
10:29:20  <andythenorth> and it just gives up if there are too many date variations :)
10:29:25  <planetmaker> :)
10:29:26  <andythenorth> also construction state aware iirc
10:29:36  <planetmaker> yeah, that's a nice cascade
10:29:45  <andythenorth> it needs splitting to use switches, instead of trying to do it all in one spritelayout thing
10:29:49  <planetmaker> you need to use extended spritesets for that with a parameter. That circumvents it
10:30:16  <planetmaker> then you just need to set a few variables and calculate an offset into the spriteset to return the proper sprite
10:30:22  <planetmaker> instead of zillion spriteset IDs
10:30:42  <Alberth> magic indexing instead :p
10:30:57  <planetmaker> yeah, currently in FIRS (and most grfs) it goes like
10:31:08  <andythenorth> planetmaker: iirc the parameter approach is incompatible with animated tiles, maybe :P
10:31:19  <andythenorth> we use the parameter for the frame iirc
10:31:22  <planetmaker> switch (var) { 1: other switchA; 2: other_switchB...}
10:31:23  <andythenorth> been a while since I wrote it
10:31:24  <planetmaker> andythenorth, it's not
10:31:33  <andythenorth> here’s an example: hide_sprite: (construction_state != 3) || (terrain_type == TILETYPE_SNOW) ${industry.get_date_conditions_for_hide_sprites(repeat.date_variation_num.index)};
10:31:42  <planetmaker> you can combine that basically
10:32:05  <planetmaker> but you might need to craft the spriteset accordingly
10:32:59  <planetmaker> like sprite = construction_state + n_construction_states*terrain_type + n_construction_states*n_terrain_types*animation_frame
10:33:58  <andythenorth> needs looked at some time
10:34:04  <andythenorth> after compiling gets faster :P
10:34:21  <andythenorth> currently it limits number of date variations, which minimises drawing
10:34:24  <andythenorth> not all limits are bad :P
10:34:25  <planetmaker> it takes less than 2:30m for a complete recompile. That's not terribly bad, really
10:34:47  <andythenorth> it’s more than my attention span :)
10:34:56  <andythenorth> I have about 4 hours a week where I can actually write code
10:35:00  <planetmaker> it's more than grfcodec -e firs.nfo (which is 3seconds).
10:35:15  <andythenorth> only a little bit more
10:35:16  <planetmaker> And 1 minute of the 2:30m is spent in make, including your python script overlay
10:35:24  <andythenorth> that is interesting
10:35:30  <andythenorth> I can look at that
10:35:42  <planetmaker> dunno which is make which is your scripts. did not check
10:35:47  <andythenorth> I can time my scripts
10:35:49  <planetmaker> the make is terribly slow really
10:36:25  <planetmaker> anyhow, I think one cannot do magic in a compiler and do that instantaneously
10:36:46  <planetmaker> it's two opposing aims which pull used time in different directions
10:38:31  <andythenorth> I would favour a bit less magic
10:38:46  <planetmaker> less magic is nfo ;)
10:39:00  <planetmaker> a linker is even more magic
10:40:06  <andythenorth> we had to get FIRS off nfo
10:40:19  <planetmaker> and the things like the snow / construction_state / terrain / date awareness in that completeness is terrible in nfo. if I remember correctly :)
10:40:25  <andythenorth> yes
10:40:29  <andythenorth> really really terrible
10:40:34  <andythenorth> that’s why we switched
10:40:43  <planetmaker> I recall :)
10:42:13  <Alberth> just write in plain grf, you're done compiling in 0 seconds!
10:42:33  <planetmaker> :D yeah
10:43:00  <planetmaker> ^D^@ÿAå^@^@µÁÿ^TCINFOTDESC^BÃ<9e>î<82><8e>FIRS Industrie-Set^Mî<82><98>Von andythenorth,
10:43:06  <planetmaker> fun times ahead ;)
10:49:41  <andythenorth> urgh
10:55:30  <planetmaker> firs.grf really has an impressive size of the action14 block(s)
10:58:29  <planetmaker> @base 16 10 C2E2
10:58:29  <Webster> planetmaker: 49890
10:58:36  <planetmaker> 50kBytes of action14 or so
11:00:04  <Alberth> woo :)
11:00:27  <planetmaker> but maybe it's wrong. Just browsing through firs.grf by hex editor
11:01:01  <andythenorth> hmm
11:01:09  <andythenorth> 3s for the python templating, seems low
11:01:57  <andythenorth> other sets are smaller than FIRS, but have a longer python step
11:04:30  <planetmaker> cpp takes a few seconds itself as well, it seems. but ...
11:07:25  <andythenorth> from an unprimed cache, the python step is 30s
11:07:32  <andythenorth> more like what I expected
11:15:57  * andythenorth has horrible ideas
11:16:18  <andythenorth> the binary output in the grf is deterministic?
11:16:40  <andythenorth> if I render the same set of action 14s, I always get the same sequence out?
11:19:11  <andythenorth> considering making a redneck linker, by brute force
11:19:46  <andythenorth> using python splits and joins :P
11:29:04  <Alberth> md5sum between grfs at different platforms says no
11:29:30  <Alberth> but whether it is purely platform, is unknown
11:31:45  <planetmaker> maybe it's also the python version (if you mean the diff I showed yesterday). It's 3.2 on linux and 3.3 on windows
11:33:11  <andythenorth> I imagine there’s some housekeeping I wouldn’t understand too
11:33:15  <andythenorth> counts of bytes or such
11:34:41  <Alberth> dicts and sets use hashing, which may be based on eg memory address of the object or so
12:15:55  *** frosch123 has joined #openttdcoop.devzone
12:23:31  *** yorick has joined #openttdcoop.devzone
12:27:02  <DevZone> Project Japanese Stations build #58-push: SUCCESS in 17 sec:
12:36:33  *** gelignite has quit IRC
13:07:51  <DevZone> Project Japanese Stations build #59-push: SUCCESS in 16 sec:
14:15:44  *** LSky` has joined #openttdcoop.devzone
14:30:42  *** andythenorth has quit IRC
15:04:10  *** andythenorth has joined #openttdcoop.devzone
15:49:50  *** andythenorth has quit IRC
16:21:59  <DevZone> Project Japanese Buildings build #217-nightlies: SUCCESS in 19 sec:
16:30:18  *** andythenorth has joined #openttdcoop.devzone
16:33:59  <DevZone> Project Finnish Rail Infrastructure - Rails build #331-nightlies: SUCCESS in 8 min 18 sec:
16:41:22  <DevZone> Project road-hog build #217-nightlies: SUCCESS in 42 sec:
16:48:15  <DevZone> Project Iron Horse build #896-nightlies: SUCCESS in 1 min 34 sec:
16:49:36  <DevZone> Project Dutch Trainset build #178-push: SUCCESS in 1 min 21 sec:
16:49:59  <DevZone> Project termite build #20-push: SUCCESS in 22 sec:
16:51:04  <DevZone> Project xussrset - Trains from Russia build #301-push: SUCCESS in 3 min 37 sec:
16:51:37  <DevZone> Project ecs build #38-push: SUCCESS in 32 sec:
16:51:39  <DevZone> Project 2ccts build #132-push: SUCCESS in 1 min 39 sec:
17:28:47  *** andythenorth has quit IRC
17:32:03  *** andythenorth has joined #openttdcoop.devzone
17:59:23  *** andythenorth has quit IRC
18:09:58  *** andythenorth has joined #openttdcoop.devzone
18:42:26  *** andythenorth has quit IRC
19:03:43  *** oskari89 has joined #openttdcoop.devzone
19:08:58  *** andythenorth has joined #openttdcoop.devzone
19:33:47  *** andythenorth has quit IRC
19:43:48  *** andythenorth has joined #openttdcoop.devzone
19:46:54  *** andythenorth has quit IRC
19:51:22  *** andythenorth has joined #openttdcoop.devzone
19:57:58  <Alberth> gn
19:58:50  *** Alberth has left #openttdcoop.devzone
20:40:09  *** andythenorth has quit IRC
21:40:36  *** KenjiE20 has quit IRC
21:41:53  *** KenjiE20 has joined #openttdcoop.devzone
22:28:01  *** LSky` has quit IRC
23:36:44  *** frosch123 has quit IRC

Powered by YARRSTE version: svn-trunk