> Is it planned that users of Eve will routinely construct new or modify existing domain-specific editors to the task at hand?
Not sure at this point. For now we are focusing on the core interactions and we'll build out the rest once we've put tools in peoples hands and seen how they work.
> What domain-specific editors are you planning to include in public release? Something for UI construction on the web?
The table, rule and function editors for managing data and logic. A UI editor, hopefully with some constraint-based layout model. Various debugging/tracing/understanding tools.
> Since first focus of Eve is on web apps (and it's hosted on javascript, correct?), will it cover both client and server applications?
It will have to. Not sure yet how that will work. We can do websocket-style stuff at the moment but we will want to provide something simpler on top of that.
The editor and compiler are mostly written in Eve. The core language is tiny but very extensible - since execution order is not specified explicitly but determined from data-flow you can drop new logic in anywhere to add new compiler optimisations or new editor commands.
The language is also very data-centric - this week I was playing with programs that modify their own code by writing to the ast tables, triggering the incremental compiler to update the dataflow plan.
> If I remember correctly, you're planning to use datalog+time+constraints and plan to use materialized views extensively, is it so? Will ability to work with changes as a stream and aggregate over them in more interesting ways than "the last always wins" be builtin in Eve?
We're still trying out different models of time. The ideal semantic model is append-only, never forget. Implementing that on a real machine needs some restrictions to prevent exploding. Dedalus (http://db.cs.berkeley.edu/papers/datalog2011-dedalus.pdf) attaches epochs to each facts and only allows rules to query the current or last epoch. Edelweiss (http://db.cs.berkeley.edu/papers/vldb14-edelweiss.pdf) allows querying any epoch but uses static analysis to figure out when data can be safely deleted without changing the results.
> Have you tried to show a prototype to children and how do they respond in learning such system if so? What was the reaction of academic friends who were not too familiar with standard forms of programming (like you did with the first prototype of Aurora)?
No children yet. We test things on adult non-programmers regularly. That lead to a couple of surprising changes like getting rid of nesting / scoping (no matter what we did visually, people just couldn't figure it out).
> If I'll ask more questions will it be all right and not too much trouble? :)
It would be interesting to hear how much of EVE code has it taken to implement compiler or editor. And is there any interesting bottlenecks you would need to optimize out before release?
And how was the experience of writing a lot of EVE code? Do you feel yourself a lot more productive? Have you become accustomed to a new way of writing software very fast and does it feel natural to you now?
Regarding models of time. I'm a bit familiar with Dedalus and I've skimmed over Edelweiss. I'm interested in how would you use Edelweiss in non-distributed settings (I assume Eve is not targeted at that kind of applications right now). Would you try to use it to maintain transactional visibility during re-flows of data across views and afterwards discarding dead data?
I remember you're mentioning datomic in a podcast, so I think you'd want to keep dead data at least for some time, to have an ability to run time-travel queries later. Is this correct?
Would it be possible for developers to control when facts should be garbage collected? For example, CORFU log (which is a good implementation of event log abstraction with flexible views above it) allows clients to control trimming http://muratbuffalo.blogspot.ru/2014/09/paper-summary-tango-... (also I've had a good experience implementing underneath a similar interface for a queue system, it lead to a similar range compression optimization as in Edelweiss).
All the details you've already mentioned make me dream at night wishing a sooner release of Eve to experiment with it. I hope your next answers won't make me kidnap you and beg for an earlier access :)
> It would be interesting to hear how much of EVE code has it taken to implement compiler or editor.
The current iteration of the editor is 350 lines of our text-based Eve dsl plus 450 lines of helper js which will eventually become our UI and IO libraries.
Stratification is about 30 lines. There used to be more going on in the compiler but now that we use a constraint solver for all the query plans there isn't much left for it to do. http://p2.berkeley.intel-research.net/papers/EvitaRacedVLDB2... has some interesting ideas that we will be exploring later.
> And is there any interesting bottlenecks you would need to optimize out before release?
We've managed to get pretty surprisingly far with really dumb code. We are halfway through a big rewrite so the current build of the editor is using a runtime that is missing half of it's features. Each table is stored in a big array, we do sequential scans for constraint propagation, the incremental evaluation has not been ported yet and we don't have any Edelweiss gc. It can still recalculate and rerender in 20-40ms. That's too slow for fluid animation but plenty fast enough button clicks etc.
The lesson here is that computers are really fast and we squander most of that speed on unnecessary runtime abstraction. I'm pretty excited to see how fast it will get when we have real indexes and incremental evaluation.
> And how was the experience of writing a lot of EVE code? Do you feel yourself a lot more productive? Have you become accustomed to a new way of writing software very fast and does it feel natural to you now?
There are ups and downs.
The main upside I've noticed is how much time I normally have to spend worrying about where to put data, how to access data, how to keep data consistent and what order to do things in. In Eve you just put it in a table and let the runtime worry about the rest. I feel like I now spend a lot more time thinking about the actual problem instead of shepherding data around.
The main problem we're wrestling with at the moment is identity. Since the editor is built in the Edelweiss style as a pure function of the input events, we have to be able to generate stable, unique ids for anything that has a continuous identity. In the Bloom model we could just generate a random UUID and store it. At the moment we build ids by mashing together strings but I feel like the language should be helping us out here.
There are also a whole bunch of problems that come down to immaturity. We don't have a debugger yet so we rely on console dumps a lot. The compiler doesn't have much error checking so bugs in the editor tend to percolate a long way before surfacing symptoms. Implementing integrity constraints is pretty high up on my todo list right now.
> I'm interested in how would you use Edelweiss in non-distributed settings (I assume Eve is not targeted at that kind of applications right now).
There is a section in Out Of The Tarpit where the author argues that the ideal way to specify an application is as a purely functional view over all the inputs it has ever received. He then concedes that in practice you are forced to throw things away to avoid running out of memory. Reading the Edelweiss paper convinced me that we could program as if we were living in Mosely's ideal world and then separately specify a garbage collection strategy. It's just another instance of separating specification from optimisation which I believe is key to managing complexity.
This is probably the most risky part of our plan but we can always fall back to the Bloom model instead. The actual language implementation doesn't really vary between them.
I find the details you are giving here interesting, and quite promising.
> That lead to a couple of surprising changes like getting rid of nesting / scoping (no matter what we did visually, people just couldn't figure it out).
I worry about that -- I can see where its probably a huge gain for initial usability but my concern with it is that it shows signs of one of the big problems with spreadsheets in that they make it easy for non-programmers to start with something, and add complexity to it until not only they can't maintain it, but calling in a professional programmer doesn't make it maintainable either, and you essentially need to do a ground-up reanalysis and rewrite to get it into a state where it is maintainable with the complexity it has grown to.
Scoping is basically a way of mapping human-readable names to uniquely addressed code objects. When I say "foo" the compiler walks up the lexical scope until it finds a declaration for a variable called "foo" and replaces the human-readable name with the corresponding unique id.
Instead of having a set of rules to map the human-readable names, we generate unique ids for everything at edit time and tag them with the human-readable name. So when you type "foo", you get an intelligently sorted auto-complete box asking you which foo you mean.
This is one of the advantages of not directly editing the program text - we can insert all kinds of useful metadata directly into the AST.
> ...they make it easy for non-programmers to start with something, and add complexity to it until not only they can't maintain it...
Like most things, the problem can't be solved with technology alone. We can improve the tools to make it easier to organise code and to make it easier to understand other peoples code but we will also have to find ways to scale up the teaching aspect. It's the 'pit of success' - we have to design the experience so that people naturally end up doing the right thing. That includes building a community where people learn from each other and are exposed to better ways of structuring their thoughts.
> Instead of having a set of rules to map the human-readable names, we generate unique ids for everything at edit time and tag them with the human-readable name. So when you type "foo", you get an intelligently sorted auto-complete box asking you which foo you mean.
> This is one of the advantages of not directly editing the program text - we can insert all kinds of useful metadata directly into the AST.
I suppose I'll have to see how it works when there is something available to poke on, but that description doesn't allay my concern about it making a programming environment that makes writing easy, but reviewing, understanding, and maintaining previously-written programs hard in much the way that spreadsheets do.
If the information needed to disambiguate references is buried behind the scenes and not readily apparent in the primary interfaced used for editing, that seems problematic.
There is also much less stuff to disambiguate. The only names you ever refer to at the moment are tables. Most databases get by just fine with no namespacing at all.
As Jonathan Edwards says in his subtext work "names are too valuable to waste on talking to compilers." Making naming more of an interface concern rather than a part of the language is a great IDE and can greatly complement the way we use lots of search to write programs anyhow.
> Most databases get by just fine with no namespacing at all.
True, but most databases are very limited components of the program(s) that use them, which do use namespacing, not the complete platform for the program.
We also get to have arbitrary names rather than trying to stuff everything into single word. "People who disagree with me on the internet" is a valid name for an Eve table.
It's a valid name, albeit on that requires quoting, in pretty much any SQL based database, too. A lot of organizations have coding standards that prohibit names requiring quotes, and/or long names, but most databases support them just fine.
That said, I'm quite interested in getting a hold of eve.
Not sure at this point. For now we are focusing on the core interactions and we'll build out the rest once we've put tools in peoples hands and seen how they work.
> What domain-specific editors are you planning to include in public release? Something for UI construction on the web?
The table, rule and function editors for managing data and logic. A UI editor, hopefully with some constraint-based layout model. Various debugging/tracing/understanding tools.
> Since first focus of Eve is on web apps (and it's hosted on javascript, correct?), will it cover both client and server applications?
It will have to. Not sure yet how that will work. We can do websocket-style stuff at the moment but we will want to provide something simpler on top of that.
> Will there be an FFI?
Yes. You will be able to add new functions / datatypes (like http://www.postgresql.org/docs/9.1/static/xtypes.html), new table/index data-structures (like http://www.postgresql.org/docs/9.1/static/gist-intro.html) and new query-plan/data-flow nodes.
The editor and compiler are mostly written in Eve. The core language is tiny but very extensible - since execution order is not specified explicitly but determined from data-flow you can drop new logic in anywhere to add new compiler optimisations or new editor commands.
The language is also very data-centric - this week I was playing with programs that modify their own code by writing to the ast tables, triggering the incremental compiler to update the dataflow plan.
> If I remember correctly, you're planning to use datalog+time+constraints and plan to use materialized views extensively, is it so? Will ability to work with changes as a stream and aggregate over them in more interesting ways than "the last always wins" be builtin in Eve?
We're still trying out different models of time. The ideal semantic model is append-only, never forget. Implementing that on a real machine needs some restrictions to prevent exploding. Dedalus (http://db.cs.berkeley.edu/papers/datalog2011-dedalus.pdf) attaches epochs to each facts and only allows rules to query the current or last epoch. Edelweiss (http://db.cs.berkeley.edu/papers/vldb14-edelweiss.pdf) allows querying any epoch but uses static analysis to figure out when data can be safely deleted without changing the results.
> Have you tried to show a prototype to children and how do they respond in learning such system if so? What was the reaction of academic friends who were not too familiar with standard forms of programming (like you did with the first prototype of Aurora)?
No children yet. We test things on adult non-programmers regularly. That lead to a couple of surprising changes like getting rid of nesting / scoping (no matter what we did visually, people just couldn't figure it out).
> If I'll ask more questions will it be all right and not too much trouble? :)
Keep 'em coming.