shezi.de

Claude Code or How I Stopped Worrying And Love the Codepocalypse

Thu, 07 May 2026 08:00:00 +0200

I've recently started to try the new hype “agentic coding“, which is something absolutely new and not simply vibe-codingFor some strange reason Mistral's implementation is called mistral-vibe. 🤷‍♂️, no no, completely new and entirely different. I've actually played around with it, using Pycharm Junie with different models, Claude Code with different models and Mistral Vibe. They're all the same, essentially, by the way, so pick whichever one you like. I even went to a workshop, organised by a client for their internal education, with two “expert trainers”, for coders and businesspeople.
I have learned a great many things from my experiences, and my professional opinion can be summarised like this:

Agentic coding goes like this: You tell it what you want, it'll go away for two to 30 minutes and then it's done. Or “done”. You can influence the level of done-ness by choosing different models and different strategies, and writing different instructions into different markdown files. It's all very scientific. The one thing you must not do, however, is to look at it. No, no, never look at the code. You may check the test coverage and the greenness of the CI system, but never the code itself. Not even the plan that you have the agent make for you, or the review document. Simply press enter and be done with it.

And yeah, sure, it somewhat works. Sometimes. Almost. But not really, actually.

Let's say that you're a carpenter and you're producing beautiful pieces of furniture with a varied selection of tools. And then someone hands you a chainsaw. Would you go: “oh, this is so much faster than what I did before” and use that single tool only for all your needs? Sure, sure, it's super quick and very easy to slice up stuff and also much less hard work than doing all the things yourself. But your furniture will be... let's say, coarse. And also not very stable. And if you want to change something, you'll just have to saw it all down and rebuild it again. And if you want to create a building with it, I wouldn't want to live in it.

But that's how y'all are going about it. You take your hard-earned, finely-tuned and well-honed skills, sharp and precise as a scalpel and exchange them for a freaking chainsaw. And the stuff you might produce might look nice at first glance, and it might work ok if you don't look at it too closely. And by “too closely” I mean “at all“.
The stuff I've seen... even the youngest learners would be ashamed to hand in that stuff. But if you say “I made that with AI”, all is fine. AS IF! And AS IF you actually had made it. You sat next to an idiot savant while they made something that vaguely looks like the thing you described, and you probably twiddled your thumbs and wasted your life at the same time, too.

Here's the thing: I can produce good software, and I can produce bad softwareAnd I specifically say “software” here. This goes beyond mere code, since it includes architecture, structure, integration, and also taste, which is the most important thing when designing APIs.. And the difference isn't super-easy to discern, if you're not a producer of software, which has been a problem for ages. “But you're almost done” is what your PM would say, because they didn't know better. And you had to educate them and push back and explain about the first 90% and the second 90%, but also about black triangles and about Mythical Man Months . But all that is gone, because now you're stuck in eternal almost-done, and

Congratulations, because that's exactly what the industry and the world needed: more bad software.

And that's not even the worst part!

But it gets worse, because you give these tools to amateurs. To the PMs and the POs and the business analysts, so they get to make bad software now. And they can't understand the difference, until it breaks and takes your carefully balanced systems down. But don't worry, because cLaUdE cAn FiX iT!
I would absolutely give a chainsaw to a beginner on their first day, sure!

I've seen people try to copy their error messages into their vibe coding interface, and when that didn't help, they tried againOk, that one's on us, we trained people to just try it again.. And when that didn't help, they installed a Playwright MCP and let the statistical parrot control their main web browser. The one that has all their logins and passwords and access to the internet, you know. Working software developers helped them do this!
WHAT THE ACTUAL FUCK? Yes, it certainly is a good idea to add some nitro to the tank of the chainsaw and also don't wear any protective clothing and also just put it on this bag of TNT when you start it up!

And now I hear people talking about the stupidest ideas like: “What can you even tell young people to learn? We won't need developers or physicists or lawyers or actors or musicians any more because ChatGPT can do all that.” These people are so mislead that they will endanger their children's futures for pure laziness. They've already lost so much mental capacity to the machines that they can't even think about what a full person could do in their life. It's so sad and stupid and also all kinds of disgusting.

But wait, the worst part is this!

To my fellow software engineers I say this: HAVE YOU NO FUCKING SHAME? Are you not ashamed to be associated with this slop, this drivel, this entire fucking mess? Are you so blinded by the allure of laziness, of not having to do the thinking yourself that you go along with this shit!

I've seen guys vibe-code some feature for their website and deploy it without looking at the code even once. Not even a glance. I am so freaking astonished that anyone calling themselves a professional would sink that low.

Actually no, it's not that anyone would do that. It's the speed with which our profession abandoned all pretense of precision or accuracy or dignity or professionalism.

You're addicts, go get help!

A tool

A chainsaw is a toolEither it is the tool or you are. Your choice., and, for the right thing, you get to use it. Carefully, and with deliberation and respect. You get to chop up stuff with it, and then you take the stuff and build something out of it.

I'm not going to pretend that AI and vibe coding is not an amazingly powerful tool. I'm not going to pretend that I'm not using it, or that I'll stop using it.

But I know what I'm doing and I'm not going to blindly apply it to everything. I'm going to have architecture in my software, because that's whats going to make it usable ten minutes from now, when you can't even remember which framework you're using because you never bothered to check.

Someone's going to need to help you when you finally find out the limits of your vibe coding agent. When your carefully constructed building comes crashing down because no one understands what it's even made out of; not your agent and certainly not you – you will need professional help. This is something I can do. Because I spent the time and learned all that stuff, and didn't spend my time sitting next to a box that pretends to do it for me.

So if you find yourself in that situation, you may talk to me about it.

None of the words in this essay were written by an AI. They were all written by RI, real intelligence.

To-Read: the books I plan on reading some day

Fri, 27 Jun 2025 08:00:00 +0200

This list will grow and shrink. Mostly grow.

War with the Newts
Monk and Robot by Becky Chambers
Atacama by Jendia Gammon
The Incandescent by Emily Tesh
A Rebel's History of Mars by Nadia Afifi
Shroud by Adrian Tchaikovsky
Of Monsters and Mainframes by Barbara Truelove

DjangoconEU2025: Talk recommendations

Thu, 05 Jun 2025 08:00:00 +0200

It looks like the DjangoconEU2025 talks are starting to appear on their youtube channel. Cool!

Of course, all the talks are great and you could, should watch all of them. What follows is not a judgement on any of the talks, especially not on the talks that I'll skip. However, some of the talks are more equal than others, and play more to my taste, so here is a list of things that I found interesting:

Day 1

Tim Bell: Turn back time: Converting integer fields to bigint using Django migrations at scale (conference page): The harrowing tale of what lengths you need to go to if you forgot to start out with BigInt fields as primary keys, and not expecting to ever have database tables with 2 billion rows.
Adam Johnson: Data-Oriented Django Drei (conference page): More about database indexes and how to use them in smart ways.
Sebastian Rode: The fine print in Django release notes (conference page): You can find the most interesting things by simply reading the Django release notes.
Aivars Kalvāns: How to solve a Python mystery (conference page): ... by diving deep into tracing the Linux system. And this was billed as Audience Level: Intermediate.
Slawa Gladkov: Logs, shells, caches and other strange words we use daily (conference page): A bit less technical but a bit more entertaining, and definitely worth watching -- even if you knew all of these terms before, of course you did!
Lightning Talks: Always good! Watch at least the ones on “Caddy snake“ (which led to heated discussions on the bus later), and the one on “safer migrations”.

Day 2

Mia Bajić: The Most Bizarre Software Bugs in History (conference page): very entertaining and thought-provoking.
Wouter Steenstra: 100 Million Parking Transactions Per Year with Django (conference page): finally some real-world data on how far you can go with a Django app, and what tricks you could/should employ
Haki Benita: How to get Foreign Keys horribly wrong in Django (conference page): Fantastic talk on the database and migrations and no, I did not know that NULL keys would be indexed!
Jacob Walls: Dynamic models without dynamic models (conference page): just the absolute madness of this scheme makes it worth watching. If you ever do something like this in production we need to have a serious talk!
Lightning talks: Always good! Especially the one on how to fall asleep within five minutes

Day 3

Graham Knapp: Feature Flags: Deploy to some of the people all of the time, and all of the people some of the time! (conference page): I've always wanted to do feature flags, and this here is one way of doing it.
Vince Salvino: One Thousand and One Django Sites (conference page): how to host a thousand pages (and more)
Agnès Haasser: Europe, Django and two-factor authentication (conference page): The challenges of implementing and shipping MFA tokens to your users. And also, you should use MFA tokens!
Lightning talks: always good!

Rounding properly in Python

Sun, 11 May 2025 08:00:00 +0200

If you want to see code, skip down to the solution.

Some time ago, Python changed its rounding algorithmAnd by “some time ago“ I mean 17 years at the time of writing. Instead of always rounding up at 0.5 values, it would round towards the even number, so 0.5 would be rounded down to 0, while 1.5 would be rounded up to 2:

>>> round(0.5)
0
>>> round(1.5)
2

The reasoning behind this is that it's conforming to the IEEE754 standard on rounding numbers, or, well, at least the specific default rule of it. The better reason is that it takes away an aspect of rounding bias. The method called “round half to even” is unbiased both on positive/negative numbers and towards/away from zero or infinity. Mathematically, it's nice.

It's just not what most people expect when rounding numbers, what they expect is “rounding as we learned it in school”, also called “commercial rounding” or “round half away from zero”. And that's what I'd like to have. Well, as one commenter put it, simply implement it by hand. Ah yes, that's exactly what I had intended for a quiet evening, re-implementing basic math functions that should be available everywhere.

Unfortunately, things aren't that simple, and for this specific operation, we cannot stay with floating point numbers, since they are encoded in binary and mixing terms like “round to the nearest decimal digit” really don't make sense in that context. So we'll have to switch to Decimals, because those can properly encode decimal numbers.

Unfortunately, it's not that easy either. Reading through the documentation in the standard library on Decimals, you might come away with this idea:

>>> from decimal import *
>>> getcontext().prec = 6
>>> Decimal(1) / Decimal(7)
Decimal('0.142857')
>>> getcontext().prec = 28
>>> Decimal(1) / Decimal(7)
Decimal('0.1428571428571428571428571429')

And you'll think to that you'll simply get the current context, set its precision to 3 and be done. But unfortunately:

>>> getcontext().prec = 3
>>> Decimal('0.12345')
Decimal('0.12345')
>>> +Decimal('0.12345')   # ok, we have to force a precision adjustment, 𝓯𝓲𝓷𝓮
Decimal('0.123')
>>> +Decimal('0.1235')
Decimal('0.124')
>>> # nice!
>>> # ... some time later
>>> Decimal('12345') + Decimal(1)
Decimal('1.23E+4')
>>> # WTF?!
>>> Decimal('12345') + Decimal(5)
Decimal('1.24E+4')
>>> Decimal('12355') + Decimal(5)
Decimal('1.24E+4')
>>> # WT actual F?

You see, that beautiful prec parameter sets the precision of the decimal type and is set to 28 by default. It specifies how many decimal digits a number carries in total. So the result of 12345 + 1 --> 12300. Also, per the last line, rounding towards even is still active. Nice!

Things I tried out

Here are six variations of how to do proper rounding that I thought up from reading the documentation. Not all of these are correct. Try to guess which ones work!

Variant 1

getcontext().rounding = mode
round(number, 0)

Variant 2

ctx = Context(rounding=mode)
dec = ctx.create_decimal(number)
round(dec, 0)

Variant 3

ctx = Context(rounding=mode)
dec = ctx.create_decimal(number)
dec.quantize(Decimal('0')))

Variant 4

ctx = Context(rounding=mode)
ctx.quantize(number, Decimal('0')))

Variant 5

with localcontext(rounding=mode):
 ctx.quantize(number, Decimal('1'))

Variant 6

with localcontext(rounding=mode):
 round(number, 0)

Solutions

Variant 1

getcontext().rounding = mode
round(number, 0)

This one works, but it sets the rounding mode globally, for all decimal processing in your program. Depending on your needs, this might be what you want. Set your rounding mode once, be done.

Variant 2

ctx = Context(rounding=mode)
dec = ctx.create_decimal(number)
round(dec, 0)

This one does not work. The function create_decimal only does its thing once, the created number does not have an association with the context.

Variant 3

ctx = Context(rounding=mode)
dec = ctx.create_decimal(number)
dec.quantize(Decimal('0')))

This one also does not work. The function create_decimal still only does its thing once, the created number does not have an association with the context, even if you use fancy words like quantize.

Variant 4

ctx = Context(rounding=mode)
ctx.quantize(number, Decimal('0')))

This one works, and it's actually my preferred solution, because we don't have to change any global state. We simply apply the context to the operation, even if it is through unnecessarily complex verbiage.

But beware of that second parameter. It does not refer to the number of digits, instead it refers to the exponent. If you want to round to one digit, that parameter needs to be Decimal('0.1') or Decimal('1e-1').

Variant 5

with localcontext(rounding=mode):
 ctx.quantize(number, Decimal('1'))

This one works. Its effects are localised to a specific context and will reset it back afterwards. Still uses fancy words, though!

Variant 6

with localcontext(rounding=mode):
 round(number, 0)

Nice, clean, localised, readable, and works. I like it. But not as much as the other one.

Rounding modes

You can read up on the available rounding modes, or you can use the little table I made. You can also check out the code I used to make this table, just in case you don't trust me.

	0.4	0.5	0.6	1.4	1.5	1.6	-0.4	-0.5	-0.6	-1.4	-1.5	-1.6
ROUND_CEILING	1	1	1	2	2	2	-0	-0	-0	-1	-1	-1
ROUND_DOWN	0	0	0	1	1	1	-0	-0	-0	-1	-1	-1
ROUND_FLOOR	0	0	0	1	1	1	-1	-1	-1	-2	-2	-2
ROUND_HALF_DOWN	0	0	1	1	1	2	-0	-0	-1	-1	-1	-2
ROUND_HALF_EVEN	0	0	1	1	2	2	-0	-0	-1	-1	-2	-2
ROUND_HALF_UP	0	1	1	1	2	2	-0	-1	-1	-1	-2	-2
ROUND_UP	1	1	1	2	2	2	-1	-1	-1	-2	-2	-2
ROUND_05UP	1	1	1	1	1	1	-1	-1	-1	-1	-1	-1

No, I have no idea why ROUND_05UP would be a good idea.

The Solution

So, here is how you round like you learnt in school, localised to a specific context:

from decimal import Decimal, Context, ROUND_HALF_UP

ctx = Context(rounding=ROUND_HALF_UP)
rounding_exponent = Decimal('1E-4')
number = ctx.create_decimal_from_float(f)
result = ctx.quantize(number, rounding_exponent)

Change the 4 to however many digits you need!

CSV sucks

Thu, 27 Mar 2025 08:00:00 +0100

I've said that CSV sucks many times.

I am on record saying publicly that CSV sucks. That video is a lightning talk that I held at Djangocon 2018 in Darmstadt, but I've said that bit before and after.

I advise against using CSV in any situation where you can avoid it. It is a bad format, it has bad usability and you shouldn't use it.

CSV is also not “dead” in any sense of the word. In fact, I would say it is undead, animated by the will of mad wizards, against its will, and it should have died a long time ago.
From time to time, I read from proponents of the CSV format, and most of the time, that's ok. You do you! Stay stuck in the stupid timeline. But sometimes, they just rub me the wrong way. Like it happened yesterday on the front page of HN. Those arguments come from an especially weird place, because they compare CSV to parquet, JSONL, MessagePack and the like, most of which have very different use cases. If you start out with a format that doesn't fit your use case, you're really quite doomed from the startI'd really like to see you encode OpenTelemetry data with nested spans and added records into a CSV file in a sane way.. If the format you choose happens to be a bad format (like CSV), you are beyond help.

Before we get to why CSV is really really actually quite really bad (it will shock you!), let me go through the presented arguments real quick.

1. CSV is dead simple

Yes, sure, except when it isn't. If everyone could just agree on what CSV actually is, then, sure, we might say that it's simple. Still not as simple as the proper solution which I will present to you at the end, but ok, simple enough, I guess. Alas, in the real world it's about as simple as travelling to another country and plugging in your electrical appliances there. Except at least the different plug standards are at least not ALL called the same thing.

Here's the documentation of the CSV reader library in Python. If you think that one counts as “simple”, well, you must be way smarter than meTo be fair, that is possible. Probable even. Alright, alright, you probably are much smarter than me!.

By the way, so is JSONL.

2. CSV is a collective idea

No, it's a collective bad idea. There are much better alternatives (see below) that are well-thought-out, a property that collective ideas famously lack.

To me, that argument boils down to: “millions of raccoons eat garbage, so eat more garbage!”

3. CSV is text

Yes, and that comes with advantages and disadvantages. So let's count that as a neutral point, and compare it with some alternatives below. By the way, so is JSONL.

4. CSV is streamable

Well, yes, maybe. If that's your use case, fantastic. There are still better alternatives, below. By the way, so is JSONL.

5. CSV can be appended to

On the off chance that I may be repeating myself, so are most of the alternatives below. By the way, so is JSONL.

6. CSV is dynamically typed

I am a fan and proponent of dynamic types, and I rather dislike statically typed programsin most situations, where appropriate, depending on use case, etc. etm., but for interfaces, I think you should have the strictest contracts you can make, and make very damn sure that the provider of the interface sticks to the contract. That's the reasoning behind all of the interface description languages, from the most ancient WSDL down to the most modern TypeSpecIT HAS IT RIGHT IN THE NAME!.

For an interface, you must make sure that what you get is what you expect.

Guess how many guarantees CSV gives you? Zero!

There is a beautiful story where you can download the historic Euro exchange rates from the ECB as a CSV fileNote that the first thing this article from CSVbase (of all places) does is convert it to a proper data format., but they mistakenly added a comma at the end of the row, and now they will have to keep that useless comma forever because it might break legacy customers' parsing, all thanks to an under-specified format language.

By the way, so is JSONL.

7. CSV is succinct

Sure, compared to formats where you can have differently-structured objects in each line where you must repeat the keys (again, use case!), CSV is slightly more succinct. Compared to other tabular formats, it is not.

Of course, statically-typed numbers could be represented more concisely, but you will not save up an order of magnitude there neither.

Go download that ECB file from above and say that again!

8. Reverse CSV is still valid CSV

This must be the weirdest argument in the bunch. You can hack your way to reading from the end of a CSV file by “feed the bytes of your file in reverse order to a CSV parser, then reverse the yielded rows and their cells' bytes and you are done (maybe read the header row before though)”?!? And that is supposed to be simpler than scanning for newlines (or any record separator) from the end of the file?! Or doing an index scan in any of the grown-up formats? I beg your hecking pardon?

9. Excel hates CSV

The argument of “interoperation with the most widely-used tool is possible, but difficult, and may break in unexpected ways” is not the own you think it is.

Signed: xan, the CSV magician

Look, this is really what it boils down to. I don't want a magician of a data interchange format. I want an interchange format to be well-defined, well-supported by libraries and languages and without surprises nor Stockholm syndrome. I don't want magic and wizardly knowledge just to transfer some rows of data.

The worst thing about it

It was worse than a crime; it was a blunder.
- Louis Antoine, Duke of Enghien

The worst thing about CSV is that it's a hack, and a bad one at that. All of the format difficulties stem from the single fact that we use characters that can appear in the data as structural elements. That means that you must escape strings, you must think about which characters to use as separators (both for fields as well as records) and you have to deal with all the things that can/must occur when someone writes wrongly, because you can't detect it right away, or at all.
Using the same characters in data as in the format is a hack. It's the first thing you think of because it's simple and stupid, and it leads to all kinds of problems down the line.
Fortunately, smarter minds have thought about this problem before and have found a much better solution: simply use different characters. Sadly, almost no-one knows about that. Here's an excerpt from the ASCII standard:

from Wikipedia: https://en.wikipedia.org/wiki/C0_and_C1_control_codes

Here's all the code you need to parse an ASV file, with ascii separated values, in Python:

raw = open("file.asv").read()

for row in file.split("\x30"):
    for field in row.split("\x31"):
        print(field)

No ifs, ands or butsThe code for the writer is equally simple and left as an exercise for the reader.. That's all, unambiguously. No escaping, no faffing, no nothing. You want to have different data in a single file? Go use the Group Separator and File Separator codes and just stuff it all in there! And done.

If only they'd read the standard.

If only they had just taken just a little time to just think about it instead of going right ahead, we'd have been spared this silly mess. But they didn't just, they won't just, and no one in the history of absolutely everything ever has just.

Alternatives

Alright, enough whining, time for some positive outlook.

Let's only look at the use-case of tabular data with relatively unchanging formats, the occasional empty fieldHow do you code an empty field in CSV?! Do you leave it empty? Is that an empty string right in the middle of your beautiful data? Or is it a null? WHO FLIPPING KNOWS?! notwithstanding. Choose differently for different use-cases: parquet or other columnar data files are for columnar data, JSONL is for varying structures in every line!

ASV

If you have the control, go for ASV! It has all the advantages of CSV, plus none of the silliness. The only bad thing is that editors don't properly show the separator codes (for viewing, a search-and-replace with newline and tab will be sufficient), nor is it super-easy to type them.

SQLite

An even better choice would be SQLite! I mean, if you don't need to rear-feed your files and then invert them, but actually no, even then, because you can do that with range queries. Sure, sure, you can't open it with every text editor, but the tooling is portable and available almost everywhereIf you have python, you can open any SQLite file.; and sure, sure, you need to learn all of these two SQL commands: select * from table; and insert into table values (...). But you'll manage, and for these herculean efforts you get proper data integrity, proper data types (if you want, SQLite can be flexible if you need it to be), multiple tables in one file, proper binary packing and random access scanning and full SQL querying.

Protobuf

On the simpler, more machine-oriented end of the range, there's Protobuf. If you have more than a few fields, the gains from not storing numbers as text is massive! For transferring some data between API endpoints, Protobuf is a great choice!

You can download OpenStreetmap data in Protobuf format.

Summary

Don't delude yourself!

CSV bad.

Other formats better.

Choose wisely!

Die Wahl

Thu, 30 Jan 2025 19:15:00 +0100

Jetzt ist es also bald Zeit für die Wahl. Es wird wohl eine der wichtigsten meines (zumindest bisherigen) Lebens werden. Schon witzelt der Postillion dass sich die Historiker freuen, live zu sehen wie es damals begann.

Ich hatte viele Worte hier geschrieben und sie dann doch wieder gelöscht, denn eigentlich bleibt nur eins zu sagen:

Wer CDU oder AFD wählt, ist mit Schuld daran, dass Deutschland und die Welt schlechter wird. Und mit solchen Menschen will ich nichts zu tun haben.

Static Full Text Search

Sat, 11 Jan 2025 11:45:00 +0100

I've decided that what this site needs is full text search. It really is essential for navigating all my exhilarating content! Check it out on the search page!

But I also don't want to move away from static hosting. I don't want to care for a search backend and data ingestion and all the stuff that comes from hosting a site. Not that I couldn't do it, and in a professional context with a site that has enough data, that is absolutely what I'd suggestAnd in case you need something along those lines you should definitely hire me!, but for my small little personal site it sounds like too much of a hassle.
So I built a static search for this site, using my site generator Heron and the JS search library Fuse.js.

If you're only interested in learning how to do this/implementing it yourself, you can skip to the tutorial part.

Heron

It really is quite simple. I added a transformer that takes the root collection (ie. all files), pulls out the original markdown and sticks it into a single file as the simplest JSON I could think of. I didn't even bother filtering anything, all of the site is in there. Even the search page itself. You can see that file under search_index.json, and it just contains a text-only version of all the content on this site. Or any Heron site that uses the newest version of Heron and has search-index enabled, for that matterThere must be millions out there!.

Using this functionality is as simple as passing --generate-search when generating the siteRemember to remove the cache before doing that, otherwise it won't pick up the correct search contents. Bonus fun fact: it takes about 15 seconds to regenerate this site completely without the cache..

Took about an hour to write and is something like 50 lines of python. It's on the repository, in case you want to have a look.

Ugh!

Javascript

The more involved part is actually the Javascript implementation here, even though the end result isn't super complicated either. The most extravagant part is the timer that waits a bit before it does the actual search (and that isn't even needed, really).

I tried Fuse.js but the search results were not good. Then I tried uFuzzy but that one can't search in objects. It does have a great comparison page, however, and that brought me to MiniSearch, which is what I'm using now.

Let's go through the sources quickly, just so I don't forget how I did this:

<!-- loading MiniSearch from its minified single-file-script -->
<!-- of course I vendored the file -->
<script src="/js/minisearch.min.js"></script>

<!-- it's just an input field -->
<!-- you may ignore these styles, I just added them here because my CSS does not include styling for inputs -->
<input
         id="searchfield"
         type="text"
         placeholder="Search..."
         style='width:100%;padding:0.4em;font-size:21px;font-family:et-book,Palatino,"Palatino Linotype","Palatino LT STD","Book Antiqua",Georgia,serif;'>

<!-- preparing some elements for display
     since I am not using any template libraries, I have to do this manually. It is not so bad,
     because the content is small enough to model with regular fields with `id`s -->
<div id="results" style="display: none;">
    <h2 id="result_header"></h2>
    <ul id="result_list">
    </ul>
</div>

And finally, this massive Javascript script:

// two convenience functions; not strictly necessary, but make the structure below a bit nicer
function show(element) {
    element.style.display = 'block';
};
function hide(element) {
    element.style.display = 'none';
}

// once we have finished searching, we need to show the results, this function isolates that
// could be inlined, but meh!
function renderResult(result, searchPhrase) {
    let rendered = "";
    for (const item of result) {
        rendered += `<li><a href="${item.url}">${item.title}</a></li>`
    }

    result_header.innerHTML = `Results for “${searchPhrase}”`;
    result_list.innerHTML = rendered;
    show(results);
};

// these are the search options; I arrived at those through rigorous scientific experiments!!!
const miniSearch = new MiniSearch({
    fields: ['title', 'content'],
    storeFields: ['title', 'url'],
    searchOptions: {
        boost: { title: 2 },
        prefix: true,
        fuzzy: 0.3,
    },
})

// this is where the actual magic happens: first we get the search index, extract its JSON and finally...
fetch('/search_index.json')
    .then((response) => response.json())
    .then(data => {

        // once we have the data, we build the search index (this only happens once at the beginning)
        miniSearch.addAll(data);

        // this is called from the timer every time we do an actual search
        // note me cleverly putting the function in this closure!
        function search() {
            const results = miniSearch.search(searchfield.value);
            renderResult(results, searchfield.value);
        };

        // here's the only little bit of extravagance I made for myself: the timer
        // the idea is that on every input event, we re-start a timer; only when
        // the timer expires do we actually do the search. This makes the site
        // not lock up while the user is still typing.
        // Turns out, the actual search is fast enough, so I set the delay to 0; this
        // at least takes the search out of the event loop.
        let timer;
        searchfield.oninput = function () {
            clearTimeout(timer);
            if (!searchfield.value) {
                hide(results);
                return;
            }
            timer = setTimeout(search, 0);
        };
    });

This is the readable version. For production I minimised the script. By hand, of course.

So, what did we learn today? A few things!

Modern JS has a very nice for (const x of y) loop.
Modern JS has proper string interpolation. When did that land?!
Modern JS allows you to use elements directly from their id. You can still do document.getElementById and this will probably be faster if you do a lot of accesses to the same document, but for me it was just not necessary.
You can absolutely build interesting and useful things without JS frameworks and libraries. Just start building, and in less than a hundred lines of code you'll have created something nice!

Book of the Year (updated)

Thu, 01 Feb 2024 20:00:00 +0100

Every year, I find some book that's extraordinary, or different, or great, or just plain fun. This doesn't mean that the books came out in that year, just that I found them that year. I've not collected them formally up until now, so I'll have to catch up now.

The following list is updated to include the years in descending order, with the newest entry on top.

2023

Those Fascinating Numbers by Jean-Marie De Koninck

Not the newest book on this list, and not quite easy to get, this one is an absolute banger.

From the fantastic chapter structureOne per number, most often less than a page through the 20-page introduction and symbol definition, and the 209-entries-long references section, to the list of prime numbers less than 10000; this book has it all.

Even though it skips a few numbers before reaching its final destination at $10^10^10^34$, this is an absolute must-read.

2021

Immun by Philipp Dettmer (english version)

The graphics of "kurzgesagt" explain in detail one of the most complicated machines we've found in nature: the human immune system. Not only is this very relevant today, it's also super-interesting and presented in an easy-to-digest manner and with humor.

2020

We have no idea by Jorge Cham and Daniel Whiteson

So you think you know anything? Wrong! We know a small handful of things about "normal" matter, which just makes up at most 5% of all matter/energy in the universe. The rest? We don't even know what the rest is. So, no we have no idea.

The only bad thing about this book is that, after a while, you'll know that you have no idea. And the book will tell you again. And again... and again...

2019

How To Invent Everything by Ryan North

Just imagine you're on a leisurely trip through time and your time machine breaks. What do you do now? Well, the first step is to find out which time period you're in. Hopefully, people are already around, because then step 2 will be much simpler: just re-build civilisation. For that, you'll have to invent everything. Fortunately, that's much simpler than you'd think, because humanity took looooong breaks between the imporant steps. This book will give you the quick course.

2018

S. by J. J. Abrams and Doug Dorst

This is a weird and wonderful book, because while the book contains a rather weird story, that's not the main thing. The main thing are the notes that are in the margin, where two people talk about the book, and find some great conspiracy around its author. The main event is not the story in the book, it's the story around the book, and the huge number of things that they put into the book. At one point, there's a paper towel with notes on it, between two pages. That kind of effort is just amazing, and the book is amazing.

Exploring all the concepts

Fri, 02 Jun 2023 20:00:00 +0200

Every time I invent something new, some person on the internet has already made a classification of all the possible ways this could go.
- Thomas Edison, probably

We humans are astonishing collectors and explorers. Once we seize on a concept or an idea, we categorise until every last nook and cranny is catalogued.

Here are 10000 types of beetles.
Here are 1300 pages of plant species in their 96th edition, organised in such a ridiculous fashion it'll take you three hours to find the one you have in your handI had to endure this with a former girlfriend, and for some reason she just wouldn't listen to me trying to organise her plant taxonomy differently.. The book is so old you can read it on Gutenberg.org
Here are 11 different ways to list the stars in the sky, plus nine more specialised catalogues.

But we also do this to the things we make up ourselves. Here are catalogues of cars, of car parts and of car catalogues.
Here are databases for world facts, landmarks, foodstuffs, football, artwork and of course, databases, databases and databases.

And also we catalogue art and fictional things, like movies, songs or story characters.
And finally, we also do this to concepts, like in maths, physics, or magic and board game mechanics.

Which is why it is exciting when someone tries to push the boundaries of some of these categories. I know I've tried with our boardgame. And, sure, it is easy to create new implementations of a story character, or car, or country, or board game mechanic. But to define a new category, a new mechanic itself, that is very, very hard.

I was so pleased when I found three entries in a discussion on games played by aliens: Notes on card games played by aliens, Rarely seen game mechanics and Notes on rarely-seen game mechanics. And while these articles are somewhat entertaining, they don't really contain "rare" or new, or alien mechanics or ideas. Sure, chopping off a finger to make the opponent take a move back is audacious and well worth a short story, it's not really a new game mechanic, just a more extreme version of Advantage Token, or possibly even really just a regular game economy, though spread out over the meta-gameI am however tempted to mention Hand Management and I Cut, You Choose. The other things they discuss are all just instances of well-known mechanics, all of which appear in a multitude of games. They are just "rare" if you're unfamiliar with board games in general.

All of this is just a long-winded way of saying this: if you have invented, or want to invent, or are about to invent a new game mechanic, please please please let me know!

“No New Tabs!” A Firefox extension

Fri, 31 Mar 2023 20:30:00 +0200

There are some sites that will just open each link in a new tab. And I hate it.

I hate that they content-optimized themselves to the point where squeezing out another tenth of a percentI imagine. of user retention is worth annoying me with that nonsense.

So I had to take a radical step: I searched for an extension that removes that attribute from all links everywhere. And I didn't find any. But, can you imagine, there are extensions that do the opposite, open every link in a new tab.

So I had to take a radical step: I modified one of these extensions to do what I wanted. Thank you, ahgolden, for writing the best extension template ever.
So here it is, No New Tab!, and here's the source code, with all of its glorious four lines of codeWell, if you count a single } as a separate line.

And that's all.

One more thing

Oh wait, there is one more thing: I tried building this for myself and then just installing it. No need for anyone else to know about my superpowers. But Firefox won't let me install the extension, since it's not signed. Even if I disable the xpinstall.signatures.required setting -- because that one only works in Firefox Developer Edition. I installed FFDEV and tried out the extensionIt works., then submitted for signing. You can choose between “self-distributed” and “hosted”, but I didn't want the hassle so I let them host it for me. Lots of scary language around “we might check and ban your extension at any time!!!” Well, go ahead, review my four-line extension!
Naturally, I made a mistake during the upload and had to upload it again. With a new ID, but ok. It also needs an ID, which I found out from some forum thread where someone had the same problem.

So in a way, you can use my extension as the absolute minimum of what an extension needs.

Why static site generation?

Tue, 10 Jan 2023 20:30:00 +0100

I was recently asked Why a static site generator? Especially since I can work fairly well with Django. So, fair question.

When I start programming something that I want, I'm not in "professional programming" mode, but instead in exploratory mode. That, too, is something I talked about in the podcast. This means that I rarely plan ahead for very long, instead I try to reach concrete goals as simply as possible. You know, "agile".
This means that I don't care so much about doing it "right", but instead about doing it "fast". So here's the first answer to the question: When I was re-doing my blogAgain, this was the simplest way to do it. I had already had a static site generator that didn't do what I wantedOr rather, that I couldn't get to do what I wanted. Failure here is my own, not that of the software. so I replaced it with a different SSG: mine.
And that is perfectly fine for exploring and learning new programming stuff. I'm doing this for my entertainment, after all.

But there is a more general answer to the question "why static site generation", and like all software development, there are trade-offs to be made. The short summary is this: * minus: I'm not an expert * minus: less library support during development * minus: no dynamic user interface * minus: no dynamic content * plus: exact points in time for updates * plus: simple hosting * plus: more security (due to simple hosting) * plus: fast responses (due to simple hosting)

Let's take them one after the other.

I'm not an expert.

This is mainly a minus, apart from the exploratory/learning angle I mentioned before. But in general, one should try to do things that one is good at, because the expected result is better. With this, my first instict of course was to do something in Django. I know Django, and I'm very good with it. So I know that I could to this, and I could do it well.

But it's also kind of boring: another CMS, yay. So not exactly the greatest thing to do in my less-than-reams-of spare time.

So in my situation, the "new thing"-angle was more important. But this is something that you need to be aware of when designing a software solution: are you doing this because you know what to do, or because it's new and shiny and different, or because it's the best fit? In a professional setting, one should always go for the best fit.

Less library support during development

Let's face it: there are many libraries in many languages that allow you to do dynamic web applications. It's an interesting space and some of the solutions are absolutely fantastic. So one thing you can do is just choose one, follow the tutorials or your expertise and create something excellent.

Great!

The library situation for static site generation is... uh... sparse. I don't know of any. Didn't search either, but I wouldn't even know what to search for. So, there's much less guidance when creating an SSG instead of a dynamic web site.

And this is an important point: while it's often great fun to do something different, it's not always advisable. There are good reasons for the prevalence of dynamic web sites, and you should choose the right technical architecture for the problem you're solving.

No dynamic user interface

This is kind-of obvious. A static site will not have a dynamic user interface. No admin interface. No forms. No built-in way to see and edit the content.

If you need that, you need to find a different solution. Either by going fully dynamic, or maybe a sort-of hybrid solution with a dynamic editing interace and then a static siteHere are two options: Wagtail Bakery, Lektor.

No dynamic content

This is the most obvious difference between a static site and a dynamic site. One is fixed and doesn't change, the other is dynamic and can change on every request.

In case you might have missed this: that's the point about dynamic web sites. They are dynamic, d'uh!

And if you need dynamic content, that's really about the only thing you can do. Have a database, store your data there, build the site "on the fly" when someone requests it. Fine!

But do you really need it? Sometimes you might not need it.

Exact points in time for updates

You know that one of the hardest problems in computer science is cache invalidation. So with a dynamic site you go full in one direction: no cache at all, so nothing to invalidate. Easy and always fresh!

But that's usually not enough. In one of my first commercial projects we had to interact with an API that would give us product prices. Since we had about 20 products per page, we'd have to call the API 20 times, and that was just too slow. So what did we do? Yes, we stored the prices in our own memory, in a memcached. Super-fast, but: how long are you supposed to keep these prices? At what point in time do you renew them? What about products going out of stock. We needed some cache invalidation and, honestly, that was the hardest thing about the entire project.

This is a rather more general process: you have a dynamic site, it's too slow, you put some caching in front of it (the Django cache, a memcached, squid, etc), doneSee how many cache servers you can spot in the Wikipedia server infrastructure!! But, not really! Now you need to keep your caches fresh. You want your content to respond dynamically, that's the point of your site, so you must make sure that the caches aren't stale. And thus, cache invalidation.

But what if you look at the problem from the other side. What if you start not from a continually changing site and delete cached content that has changed, but instead start from a constant, unchanging site and add singular points in time when the content can change. So whenever some content changes, you push the change into the site, instead of pulling from a dynamic data set into your cache.

And thus, static site generation. I have a set of data (in the form of markdown files, but could be anything), and whenever I am happy with the state that they're inI mean, except when I make a mistake and publish a draft., I regenerate everything and push it live.

There are now exactly-defined points in time when my site changes, and that fits my use case perfectly.

Simple hosting

Not having to regenerate the site whenever it is requested also makes serving the site much simpler. There's a directory of files, go serve that. Any webserver can do that. No need for complex infrastructure, reverse proxying, app server, database, media and static serving; just simple, easy file hosting.

Heck, it's so simple, you could put it on an S3 bucket and call it a day.

Heck, it's so simple, Github will do it for you for free!

With simple hosting come other benefits. You don't need to worry about someone breaching your database because there is no database. Remove all nginx plugins and you'll have a digital fortress as secure as it can be.

And it'll be fast, too. Simple file serving is fast on its own, but if it's not fast enough, put your site on a ramdisk and you'll saturate all the networks you have. My fully generated site is about 12MB large. I can afford that much ramdisk.

in Summary

Look, I just wanted to do this, ok? It works for me, and I'm having fun.

Are there more reasons needed?

Art

Sat, 10 Dec 2022 15:31:00 +0100

(note: this is a work in progress, I'm still thinking about it)

“The anthropologists got it wrong when they named our species Homo sapiens ('wise man'). In any case it's an arrogant and bigheaded thing to say, wisdom being one of our least evident features. In reality, we are Pan narrans, the storytelling chimpanzee.”
― Terry Pratchett, "The Globe"

I was wondering why we do things. Like, sure, we need food and shelter to survive, and probably a few other basic necessities. And if those aren't met, at least to some degree, few other things are important.

But that's not what we strive for most of the time, is it? What we strive for isn't just food and shelter, it isn't just survival, it isn't just belonging-to-a-group-because-that-increases-chances-of-survival.
Why do people read books? Why do we listen to music? Why do people go to galleries and look at paintings? Why do we sit around camp fires and tell stories? Why do people risk their lives to go to the theater and the opera and meet people and sing and dance?

I want to go a step further than pan narrans, because I believe stories aren't the ultimate goal. The ultimate goal are feelings. Emotions.
We are the feeling ape, simia sensibilis.
We need emotions so much that we'll spend considerable resources to experience them. We crave emotions so much that we'll destroy our lives to get them, to feel them. We'll hurt other people, and ourselves, to safeguard "the right emotions", and we'll expend all our energy to turn "bad feelings" into "good feelings", even to the detriment of all the world.

Stories are just one way that we make others feel things, even though a very strong and durable way. We tell stories so others can experience feelings "second-hand". We invent stories so we can show other people emotions that they wouldn't think of themselves. We read books because someone told us that they liked to feel the emotions wrapped into the words.
But there are so many other ways to achieve feeling transfer: painting, music, dance, food, rituals, social groups, personal relationships. Do you think it's a coincidence that all human cultures have these? And cherish them? And develop them and refine them, and defend them against outsiders and change. It is because we crave feelings.
Our rational thinking abilities, that we are so proud of, are just a mechanism to get at more feelings, and more variation in feelings.

And thus, Art.
Art is making other people feel feelings. Stories are Art, paintings are Art, music is Art, dance is Art, food is Art. Anything that one does to make other people feel something is Art. And it doesn't matter how big or small, or for how many or few people the Art is for.
If you make a movie and it reaches a billion people: Art.
If you prepare a sandwich for your friend: Art.
If you write the best symphony in the world and are on all the news: Art.
If you dance with someone: Art.
If you sing a tune to yourself, just for yourself: Art.
If you create the worlds best-selling computer game: Art.
If you create a computer game but don't show anyone: Art.

Every one of us is an Artist. It is what we are.

Do not let anyone tell you that it's not Art you are creating. Do not let anyone tell you that your Art is bad, or that someone else is better, or that you shouldn't do Art. As long as anyone is feeling something over the things you do, even if it's just yourself, it is Art as much as anything else.
You do your Art and you be proud of it.

But there is also another side to it. We do not simply crave good emotion, we crave any emotion, and the more variety there is, the better. That also means that we need fear and tension and sadness. How boring would stories be if they were only ever happy.

And also note that Art has nothing to do with money. You can make money without doing Art and you can do Art without making money.

Every one of us is an Artist. It is what we are. You do your Art and you be proud of it.

All the best paths

Mon, 03 Oct 2022 20:00:00 +0200

When you go from A to B on a map, there is usually a best path to goI mean, that's what satnav is for, right?. But since there are many roads, there must be points where there are two paths that are optimal.

I have this silly idea that I want to find these watershed points and draw a funny little map showing the hills between them. In this post, we'll start looking at OSM data and play around a bit to find the watersheds.
This isn't a finished thing. I played around until I didn't want to any more. What follows is part of a Jupyter notebook in which I tried getting somewhereGetting somewhere! Do you get it?.

The first thing we need to do is get some OSM data. I used the excellent BBike OSM extractor and found me some Düsseldorf.

These are the libraries we're going to use:

from collections import Counter, defaultdict
import tracemalloc

import svgwrite
import pyproj
import numpy as np

First, we're going to read the map data, in its entirety. Note that this may take a little while. Depending on the data file, this might be between 2 and ∞ seconds.

Just out of interest, we're also going to tracemalloc it, just to see how much memory we're using.

filename = 'planet_6.74867,51.21731_6.79149,51.23367.osm.geojson'
bounds = (6.74867, 51.21731), (6.79149, 51.23367)

home = (bounds[0][0] + bounds[1][0]) / 2, (bounds[0][1] + bounds[1][1]) / 2

%%time
import json
tracemalloc.start()
with open(filename) as mapfile:
    mapdata = json.load(mapfile)
print(tracemalloc.get_traced_memory())
tracemalloc.stop()

(69823463, 92864418)
CPU times: user 1.7 s, sys: 98.2 ms, total: 1.8 s
Wall time: 1.79 s

700MB of allocations from an 11MB geojson file. Nice!

I flailed around a little bit, trying to find out how to work with the data. I'm only showing a little bit here, just so you can see how professional I am.

features = mapdata["features"]
len(features)

Counter(f["geometry"]["type"] for f in features)

Counter({'Point': 10186, 'LineString': 11594, 'MultiPolygon': 8044})

points = [f for f in features if f["geometry"]["type"] == "Point"]
len(points)

line_strings = [f for f in features if f["geometry"]["type"] == "LineString"]
len(line_strings)

multi_polygons = [f for f in features if f["geometry"]["type"] == "MultiPolygon"]
len(multi_polygons)

Alright, we need to go a bit deeper into the properties:

props = set()
for f in line_strings:
    props.update(set(f["properties"].keys()))

types = Counter()
highways = Counter()
junctions = Counter()
for f in line_strings:
    types.update([f["properties"].get("type")])
    highways.update([f["properties"].get("highway")])
    junctions.update([f["properties"].get("junction")])
types, highways, junctions
# turns out, only highways are interesting
highways

Counter({'trunk': 32,
         'secondary': 287,
         None: 8421,
         'footway': 1039,
         'residential': 309,
         'tertiary': 124,
         'track': 19,
         'path': 370,
         'unclassified': 4,
         'living_street': 10,
         'cycleway': 14,
         'steps': 233,
         'service': 511,
         'pedestrian': 125,
         'primary': 11,
         'secondary_link': 4,
         'platform': 19,
         'trunk_link': 48,
         'elevator': 6,
         'tertiary_link': 3,
         'corridor': 5})

We found out that only highways are interesting. You know, we could have looked at the documentation, too!.

These are the roads we are going to work with from now on:

highways = [f for f in line_strings if f["properties"].get("highway") in (
    "motorway", "motorway_link", "primary", "secondary", "trunk", "tertiary", "unclassified",
    "residential", "living_street", "pedestrian",
    "primary_link", "secondary_link", "tertiary_link",
    "bridleway", "platform",
    None,
)]
len(highways)

Since we want to produce beautiful maps (or any maps, actually), we're going to need to be able to draw them. We do this by creating SVG files. These can be zoomed and printed and edited, as well as shown inline.

Here's the utility function we're going to use. Note again how professional this is.

def draw_map(roads, points, segments=None, home_point=None):

    if segments is None:
        segments = []

    WIDTH, HEIGHT = 350, 140

    dwg = svgwrite.Drawing('svgwrite-example.svg', size=(f"{WIDTH}mm", f"{HEIGHT}mm"))
    transg = dwg.add(dwg.g())
    g = transg.add(dwg.g())

    if points:
        bounds_points = (min(p[0][0] for p in points), min(p[0][1] for p in points)), (max(p[0][0] for p in points), max(p[0][1] for p in points))
    else:
        bounds_points = (100000000, 100000000), (-100000000, -100000000)
    if roads:
        bounds_lines = (
            (
                min(min(p[0] for p in g["geometry"]["coordinates"]) for g in roads), 
                min(min(p[1] for p in g["geometry"]["coordinates"]) for g in roads), 
            ), (
                max(max(p[0] for p in g["geometry"]["coordinates"]) for g in roads), 
                max(max(p[1] for p in g["geometry"]["coordinates"]) for g in roads), 
            )
        )
    else:
        bounds_lines = (100000000, 100000000), (-100000000, -100000000)
    bounds = (
        (
            min(bounds_points[0][0], bounds_lines[0][0]),
            min(bounds_points[0][1], bounds_lines[0][1])
        ), (
            max(bounds_points[1][0], bounds_lines[1][0]),
            max(bounds_points[1][1], bounds_lines[1][1]),
        )
    )
    max_y = bounds[1][1]
    scale = min(bounds[1][0] - bounds[0][0], bounds[1][1] - bounds[0][1])
    PT = scale / WIDTH

    # g.translate(-(bounds[0][0] + scale/2), -bounds[0][1] - scale/4)
    # scale /= 2
    g.translate(-bounds[0][0], 0)

    transg.scale((1.4 * WIDTH) / scale, (1.4 * WIDTH) / scale)
    # transg.scale(0.01, 0.01)
    innerg = g.add(dwg.g())
    innerg.scale(1, -1)
    # g = innerg

    g.fill("none")
    g.stroke(color=svgwrite.rgb(200, 200, 200), width=scale / WIDTH, linecap="round", linejoin="round")

    g.add(dwg.rect([bounds[0][0], 0], (bounds[1][0] - bounds[0][0], bounds[1][1] - bounds[0][1]), fill='white'))

    strokes = {
        "motorway": {'color': svgwrite.rgb(100, 10, 16, '%'), 'width': 2 * PT},
        "raceway": {'color': svgwrite.rgb(100, 10, 16, '%'), 'width': 1.5 * PT},
        "primary": {'color': svgwrite.rgb(10, 10, 80, '%'), 'width': 1 * PT},
        "secondary": {'color': svgwrite.rgb(30, 30, 30, '%'), 'width': 1 * PT},
        "tertiary": {'color': svgwrite.rgb(80, 80, 80, '%'), 'width': 1 * PT},
    }

    for highway in roads:
        # print(highway["geometry"]["coordinates"])
        # print(strokes.get(highway["properties"]["highway"], {}))
        coords = highway["geometry"]["coordinates"]
        coords = [(c[0], max_y - c[1]) for c in coords]
        line = g.add(dwg.polyline(coords))
        if stroke := strokes.get(highway["properties"].get("highway")):
            line.stroke(**stroke)

    for p1, p2, color in segments:
        p1 = p1[0], max_y - p1[1]
        p2 = p2[0], max_y - p2[1]
        s = g.add(dwg.line(p1, p2, stroke=color))

    CIRC_R = 0.5 * PT
    for (point, color) in points:
        point = point[0], max_y - point[1]
        color = color if isinstance(color, str) else svgwrite.rgb(*color, '%')
        c = g.add(dwg.circle(point, r=CIRC_R, fill=color, stroke='none'))

    if home_point:
        print(home_point)
        point = home_point[0], max_y - home_point[1]
        c = g.add(dwg.circle(point, r=5*CIRC_R, fill='#00ffff', stroke='none'))


    return dwg

Let's draw a map of our data for the first time:

%%time
points = [
#    ((9.08915, 49.04266), (0, 100, 1))
]

highways = [f for f in line_strings if f["properties"].get("highway") in (
    "motorway", "motorway_link", "primary", "secondary", "trunk", "tertiary", "unclassified",
    "residential", "living_street", "pedestrian",
    "primary_link", "secondary_link", "tertiary_link",
    "bridleway", "platform",
)]

# bounds = (8.9178,48.9743), (9.2605,49.1111)
w, h = bounds[1][0] - bounds[0][0], bounds[1][1] - bounds[0][1]
for i in range(1, 100):
    for j in range(1, 20):
        p = (bounds[0][0] + (i / 105) * w , bounds[0][1] + (j / 20) * h), (i, j, 0)
        # points.append(p)
draw_map(highways, points)

CPU times: user 65 ms, sys: 191 µs, total: 65.2 ms
Wall time: 64.1 ms

Well that actually vaguely looks like a map of Düsseldorf.

1: move to meter-based coordinates

At some point, we're going to want to measure distances. To do so, we need to convert from WGS84/EPSG:4326 to a more well-suited coordinate system. Apparently EPSG 3035 is the way to go here, since it's accurate for all of Europe.

Or, hmm, we could also try these ones: EPSG:3068, EPSG:5679, or EPSG:7837 or any of 100 others. I don't know what the differences are and I guess they are minimal. I hope they are, anyway.

pts = points[:2]

for pt in pts:
    print(pt)
    print(pyproj.transform("epsg:4326", "epsg:3035", *pt[0]))

([6.7606583, 51.2234957], (0, 0, 0))
(-513271.50645004585, 9054181.35310964)
([6.7653554, 51.2195793], (0, 0, 0))
(-513019.06060262443, 9053634.032830823)

p = (9.08915, 49.04266)
from pyproj import Proj, transform

p1 = Proj('epsg:3035', preserve_units=True)
p1x = p1(9, 48)
p2x = p1(9, 49)

distance_m = np.sqrt((p1x[0] - p2x[0])**2 + (p1x[1] - p2x[1])**2)
distance_m

111156.71848944735

from pyproj import Transformer

transformer = Transformer.from_crs("epsg:4326", "epsg:3035")
transformer.transform(9, 49)

(-409100.56561648333, 8755738.426043686)

I have no idea what these numbers mean. But ok!

def dist(p1, p2):
    return np.sqrt((p1[0] - p2[0])**2 + (p1[1] - p2[1]) ** 2)

for hw in highways[:10]:
    coords = [transformer.transform(*c) for c in hw["geometry"]["coordinates"]]
    l = 0
    for i in range(1, len(coords)):
        l += dist(coords[i-1], coords[i])
    print(l)

602.7340984907527
160.7871203419805
91.58520679722832
34.37434665463454
11.082176061452335
115.85408874142807
150.24235157447842
65.0171774765028
456.24031388355354
110.21961116674738

Those distances look... reasonable?

Here we go, transforming our coordinates

Careful: only run this once, or you'll kill all your coordinates! Why? Because I don't want to spend another 750MB of RAM on geo data again!

# careful: only run this cell once!
for hw in highways:
    hw["geometry"]["coordinates"] = [transformer.transform(*c) for c in hw["geometry"]["coordinates"]]
home = transformer.transform(*home)

%%time
points = set()
for hw in highways:
    for coord in hw["geometry"]["coordinates"]:
        points.add((coord, (0, 0, 0)))
# points.add((home, (0, 100, 1)))
print(len(points))
draw_map(highways, [])

3458
CPU times: user 59.3 ms, sys: 3.83 ms, total: 63.1 ms
Wall time: 61.4 ms

That still looks a bit like Düsseldorf, though less so than before.

You might notice that the map isn't pointing the same way as before. Yes, well, that's unfortunate, and apparently because these projections use different norths. We won't let that discourage us...

3: the Dijkstra algorithm

Before we can start actually computing distances, we need to find a suitable data structure for our graph.

I guess, the simplest one is this: a dictionary, where the keys are nodes and the entries are lists of (other, distance). We'll have to take care that our graph is bi-directional, since we want to make sure that we can go everywhere.

def dist(p1, p2):
    return np.sqrt((p1[0] - p2[0])**2 + (p1[1] - p2[1]) ** 2)    

graph = defaultdict(list)
for hw in highways:
    coords = hw["geometry"]["coordinates"]
    for i in range(1, len(coords)):
        fr, to = coords[i-1], coords[i]
        d = dist(fr, to)
        graph[fr].append((to, d))
        graph[to].append((fr, d))
len(graph)

Ok, so dijkstra, in short: You start at a node, and you mark it as DONE. Then you visit all the nodes you can reach and mark them as active, with their current distance. Then you start visiting nodes: From the list of active nodes you select the one with the smallest distance. Mark that as DONE. Add all nodes you can reach (that are not DONE) as active with current_distance + distance. Repeat until all nodes are visited.

# meh, we'll have to find the closest point to our home so we can start somewhere!

for home_point in points:
    break
min_dist = dist(home, home_point[0])

for p in graph:
    d = dist(home, p)
    if d < min_dist:
        home_point, min_dist = p, d

home_point, min_dist, home

((-512198.0851256219, 9054105.365061712),
 16.589845930492412,
 (-512208.83020984754, 9054118.004925357))

print(graph[home_point])

[((-512192.63521925174, 9054100.065596774), 7.601697709388693), ((-512236.8403923488, 9054071.247561801), 51.63307563186496), ((-512185.5072865789, 9054116.24124847), 16.628092897033614)]

done = dict()
active = dict()

def find_minimum_active(active):
    """Find the minimum active node, remove it from the active set and return it."""
    if not active:
        return None
    pt, min_dist = None, 1_000_000_000
    for point, dist in active.items():
        if dist < min_dist:
            pt, min_dist = point, dist
    del active[pt]
    return pt, min_dist

done[home_point] = 0
for p, d in graph[home_point]:
    active[p] = d

while curr_p := find_minimum_active(active):
    # print("---")
    curr, d = curr_p
    # print(curr)
    done[curr] = d

    # print("-----")
    for next_p, next_d in graph[curr]:
        next_d = next_d + d
        # print(next_p, next_d)
        if next_p in done or (next_p in active and active[next_p] < next_d):
            continue
        active[next_p] = next_d

len(done)
# [(p, (100, 0, 0)) for p in done]

Ooh, not all points on the map are reachable from our starting point. There's actually quite a lot of roads we can't reach, mainly the southern Rheinbrücke, since the highway on-ramps are not part of my map section.

%%time
svg = draw_map(highways, [(p, (100, 0, 0)) for p in done])
with open("map.svg", "w") as f:
    svg.write(f)
svg

CPU times: user 1.11 s, sys: 10.6 ms, total: 1.12 s
Wall time: 1.13 s

All reachable points in red.

All right! We've visited all the reachable points. Now to color them!

4: determine colors based on max_distance/colormap

import colorcet

len(colorcet.fire)

We're going to use ColorCET, which is a better color map than traditional flame colors.

max_dist = max(done.values())
slot_dist = 255 / max_dist
def slot(dist):
    return int(dist * slot_dist)

swatch = colorcet.colorwheel
points = [(p, swatch[255 - slot(dist)]) for p, dist in done.items()]

svg = draw_map([], points)
with open("map.svg", "w") as f:
    svg.write(f)
svg

Rainbow-colored reachable points. Almost there!

That's beautiful. But we have not found our watersheds yet. For that, we'll need to actually draw all the paths. Any segment that is not part of a path is then a watershed. And so we need to adapt our dijkstra a little bit.

5: dijkstra with full paths

done = dict()
segments = []
active = dict()
sheds = []

def find_minimum_active(active):
    """Find the minimum active node, remove it from the active set and return it."""
    if not active:
        return None
    pt, min_dist, min_p = None, 1_000_000_000, None
    for point, (dist, prev_p) in active.items():
        if dist < min_dist:
            pt, min_dist, min_p = point, dist, prev_p
    del active[pt]
    return pt, min_dist, min_p

def middle(p1, p2):
    return (p1[0] + p2[0]) / 2, (p1[1] + p2[1]) / 2


done[home_point] = 0
for p, d in graph[home_point]:
    active[p] = (d, home_point)

while curr_p := find_minimum_active(active):
    # print("---")
    curr, d, prev_p = curr_p
    # print(curr)
    done[curr] = d
    segments.append((prev_p, curr, d))

    # print("-----")
    for next_p, next_d in graph[curr]:
        next_d = next_d + d
        # print(next_p, next_d)
        if next_p in done:
            continue
        if next_p in active and active[next_p][0] < next_d:
            sheds.append(middle(curr, next_p))
            continue
        active[next_p] = (next_d, curr)

len(done), len(sheds), len(segments)

(3202, 154, 3201)

%%time
swatch = colorcet.rainbow
# points = [(p, swatch[255 - slot(dist)]) for p, dist in done.items()]
points = [(p, '#ff0000') for p in sheds] + [(home_point, '#00ffff')]
segs = [(p1, p2, swatch[255 - slot(d)]) for p1, p2, d in segments]
svg = draw_map([], points, segs, home_point)
with open("map.svg", "w") as f:
    svg.write(f)
svg

(-512198.0851256219, 9054105.365061712)
CPU times: user 549 ms, sys: 390 µs, total: 549 ms
Wall time: 544 ms

Ah, watershed points!

And there we have it. All paths you can take in Düsseldorf, with watersheds marked in red. There are many things missing, but I stopped when I reached a nice picture.

I must have made a mistake somewhere because there should not be any closed circles. And also North should be up. And the map should be connected. And my paths should be little arrows to show flow direction. And directional roads should be directional. And footpaths should be different from highways. And the watersheds should be connected and probably hill-shaded, too. And I should show distance markers. And all of this should be much faster. And I'd like this in large-scale printable format. With rivers, and buildings, probably. You know, like in a real map.

So, lots of stuff to do next time.

Getting all mtimes in a directory recursively with Python

Sun, 11 Sep 2022 21:00:00 +0200

I need to get the mtimes for all files in a directory quickly, so I can add a good caching layer to Heron.

There's a lot of bad/beginner advice on the internet on how to do this, so I had to try it out myself. Here are three four ways to do it:

test_dir = Path('.')  

def method1():  
    total = 0  
    for dirpath, dirnames, filenames in os.walk(test_dir):  
        for filename in filenames:  
            total = os.path.getmtime(os.path.join(dirpath, filename))  
    return total  

def method2_1():  
    total = 0  
    for dirpath, dirnames, filenames, dir_fd, in os.fwalk(test_dir):  
        for filename in filenames:  
            total = os.stat(filename, dir_fd=dir_fd).st_mtime  
    return total  

def method2_2():  
    total = 0  
    for dirpath, dirnames, filenames, dir_fd, in os.fwalk(test_dir):  
        for filename in filenames:  
            total = os.path.getmtime(os.path.join(dirpath, filename))  
    return total  

def method3():  
    total = 0  
    file_paths = glob.glob("**/*", root_dir=test_dir)  

    for file_path in file_paths:  
        total = os.path.getmtime(test_dir / file_path)  

    return total  

print("  1", timeit.timeit(lambda: method1(), number=1000))  
print("2_1", timeit.timeit(lambda: method2_1(), number=1000))  
print("2_2", timeit.timeit(lambda: method2_2(), number=1000))  
print("  3", timeit.timeit(lambda: method3(), number=1000))

Here's the output for my specific test directory on a Linux computer:

  1 2.029968542000006
2_1 1.4166910230001122
2_2 1.9437335370002984
  3 3.277476520000164

Note that these are for 1000 runs each, so any of these methods will be quick enough in my case.

Using the glob-kind of method can be much faster when you can discard many files, ie. if you only need all image files or some such. I need all files here, so this does not apply to me.

I guess I should implement a version using scandir as well, but I don't expect a lot of speedup over method 2_1, since on Linux, each stat needs a stat anyway. So I'll go with 2_1 for the moment.

Headphones

Fri, 02 Sep 2022 20:00:00 +0200

I'm very particular about headphones. Not the sound, the sound is almost always fine anyway, but the fit of the phones. In the office, I have fancy over-ears that are large enough to cover my ears and soft enough so I don't have to worry about wearing them all day.
But for on-the-go, I prefer earbuds, mainly for their better isolation, both inwards and outwards. But when I'm moving, or speaking, or eating, most earbuds will fall out, or at least lose their fit so the outside world intrudes. I don't seem to be the only one with that problem.. Naturally, I want cables as well, and I prefer to have a microphone for calls.

Many years ago, I used Beyerdynamics fitted earbuds, which were perfect. They're molded to your ears by a specialist. But they're also quite expensive, and after breaking the headphones once and then losing one of the fitted pieces, I wanted a slightly less expensive optionOf course, if money is no object, there are much higher-end options..

Some years later I found a model of Soul Pulse Earbuds, and those were perfect, too. Waterproof, great fit, microphone, inverted-design so no chance of losing them, and not very expensive either. Naturally, when I wanted to replace my old ones, I found out that this model was discontinued. So after buying the last specimens I could find, I had a problem: what to do once those break.

Well, they're starting to breakor get lost, so I need to find good replacements. As any normal person would do, I ordered several different models that look good and am now testing them allYou'll notice that I won't compare on sound quality. This is because sound quality does not matter to me. They all sound equally good, or equally good enough, at least..

Amazon Basics Sport In-Ear Headphones

21.72 €

They are surprisingly not very cheap, compared to the other options in this lineup. However, they also bring with them everything that's necessary: earbuds, different ear fittings, ear wrap hoop and a carrying pouch. They come in an attractive brown cardboard box and there's an excellent manual that essentially tells you to put them into your ear.

Fit is good, not great, but I like them. They are very round, so I think I could wear them for a long time. Having worn them for a while, I noticed that they are actually too round, so they slip out of my ears all the time. Not exactly what I was looking for, and much worse than the JLab ones, which are rubbery and well-fitting.

There has come up a sound issue, actually. There's a slight rattle in one of the headphones. I'll have to wait a while until I'm sure it's recurring.

Avantree TR504 Wired Earbuds For Sports

16.99 €

I am very positively surprised by these. The cheap-looking site turned out a great product with excellent fit. They're quite bare, however: there is literally nothing in the box apart from the headphones. If you need different rubber earpiece-fitting-thingies, you better have some left from other headphonesAs you'd expect, I have a collection of about 100 of them..

Right now, I have also no idea where I put them... soo... good start?

They're also very round and comfortable to wear for a long time. And, so far, they managed to survive in my jacket pocket for some time, which is an important benchmark.

Finally, they are also blue.

I must say, as I'm wearing them more, I like them less. The very round form makes them slip out of my ears easily. So, not recommended.

JLAB JLab Fit

19.99 €

These ones are very branded, even on the rubber ear tips themselves. I'd thought that the rather sharp edges of the logo would be a problem, but so far, it's not. The added-logo-studdedness is balanced by their full rubber enclosure which I like very much. Unfortunately, they feel a tiny little bit too large for me, so I'm not sure I'll be able to wear them for a long time. There is a second set of rubber, uh, wrappings, maybe these will help. Funnily enough, the rubber covers don't fit the headphones very well, but they do fit my ears very well. They stick in and don't fall out, so that's nice. The rest of the assembly is a bit flimsy, but that's ok.

MEE audio, EP-M6PROG2-BK M6 PRO

49.99 €

The most expensive headphones in this lineup mark themselves as "in-ear monitors". The sound is brilliant, just like for each of the other ones. They have great features, like the pouch they come with, or the replaceable cablesnot that I'd know where to get such a specific cable without the headphones, or the second non-microphoned cable, or the included 6.3mm adapter.
The real gimmick however are the included ComplyFoam tips, which sounds like an excellent (and expensive) idea. And I'd say it is. The tips stay in my ears, they shield sound excellently and they don't become very uncomfortable during longer use. So recommendation on this one.

However, the actual headphone fit is not as good. The earpieces have rather sharp edges that become unfomfortable over time, so I'll have to give them a bit of a sanding. Also, they tend do slip out. I'm not sure I like them well enough for their price.

MEE Audio M6P2-WT G2

14.95 €

They are white. They fit ok. They sound ok.

I think I don't know where I put them, already.

Vivanco Sport Outdoor

14.00 €

I didn't even order this one right away, because there are so many other good ones. And the reduced price is not a good sign, given that I don't want to buy up the last few items again.
But now I've tried them and they are really good. Rubber coating all around, strong hang wire at the back, great fit for my ears. They even have volume buttons on the cable. They look quite silly, because they're much heavier than the other ones, but I don't really care about that.

Summary

In summary: I am surprised and astonished by the wide selection of great products, all of which fulfil my requirements. None have fatal flaws, except maybe the edgy pro onesI am referring to the price, naturally.. Most of them are fine, some are better, some less. The most important thing about them is still the fit, so they won't fall out of my ears, and there are some great options there.

For the foreseeable future, I'll have no problems with headphones, and that makes me happy.

Antheon — a game concept

Thu, 18 Aug 2022 00:00:00 +0200

A game conceptI used to hoard game concepts. I wrote them down in books and on pages and in documents. All with the expectation that I'd do them, some day, when I'd be ready. Well, I don't believe that any more, and also I can produce many more game designs than I can execute on. So, here we are..

This game simulates an ant colony managed by the player. The ant colony is in conflict with the elements, with scarcity, and with other colonies. The player manages the colony by marking out pheromone trails (ie. work orders) and some internal settings of the colony, like rates of workers/warriors or desired fill states for reserves. As such, there is a split between the internal management of the colony, the underground part where the queen lives; and the outside world, where workers fulfill orders. As the game progresses, the gameplay changes from micro-level management (eg. "where to place this trail" or "where to build this room") to more macro-level decision (eg. "where to start a new hive" or "which armies attack where"). Naturally, the player can go back to the micro-level, but doesn't need to. This avoids both tediousness as well as "load-breaking"This is a phenomenon I see in many progressive strategy games: at some point, the workload of all the aspects of the game become so large that the player is bogged down in detailed tactical planning instead of looking at the larger strategic picture. in the late game.

Gameplay

The main gameplay loop consists of laying out pheromone tracks containing work-orders for the workers. The ants then self-organize to fulfill the work orders. Work-orders can be of different strongness; one order might be "collect food exactly here"; another order might be "collect food in this area until exhausted"; another order might be "explore in this direction". Work orders might also fade over time, unless reinforced by the player or by the ants themselves.
Once the game progresses past a certain size, the player can give more "wide" orders, like "build a hive here" or "move an army here".

The secondary loop is the internal management of the hive. The inside of the hive is built in the same way as the outside; however, there are more abstract aspects of the colony: mainly the numerical aspects of the ant economy: storage, reserves, resource allocation, rates of production etc. Naturally, these can be set on the individual buildings, but also for a hive in total, and for the colony in its entirety.

Inspirations here could be Dwarf Fortress, Rimworld or Planetary Annihilation (for scale and army management).

Multiplayer

The game could offer multiplayer, both competitive as well as cooperative. Multiple players managing a colony shouldn't be very problematic.

Scenarios

The game allows sandbox play, competitive play (ie. sandbox with restrictions) and campaign mode.

A game in sandbox mode may enable and disable any and all mechanics that the player wants.

The campaign sees the player build a megacolony spanning an entire continent.

Graphics

The game is presented in 3D from an overhead view, with slightly stylized graphics. Zooming out will present the game state in a more abstract way.

Inspirations can be Planetary Annihilation (again, for scale and presentation) or Spore (for scale and levels of management).

License

You may use this concept in any way you see fit. If you want to work with me on this, or if you need specific license terms, email me. Otherwise you may choose to receive the text of this article under the terms of the Blue Oak Model License 1.0.0 as written here:

Blue Oak Model License

Version 1.0.0

Purpose

This license gives everyone as much permission to work with this software as possible, while protecting contributors from liability.

Acceptance

In order to receive this license, you must agree to its rules. The rules of this license are both obligations under that agreement and conditions to your license. You must not do anything with this software that triggers a rule that you cannot or will not follow.

Copyright

Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it.

Notices

You must ensure that everyone who gets a copy of any part of this software from you, with or without changes, also gets the text of this license or a link to https://blueoakcouncil.org/license/1.0.0.

Excuse

If anyone notifies you in writing that you have not complied with Notices, you can keep your license by taking all practical steps to comply within 30 days after the notice. If you do not do so, your license ends immediately.

Patent

Each contributor licenses you to do everything with this software that would otherwise infringe any patent claims they can license or become able to license.

Reliability

No contributor can revoke this license.

No Liability

As far as the law allows, this software comes as is, without any warranty or condition, and no contributor will be liable to anyone for any damages related to this software or this license, under any kind of legal claim.

as_view: a dirty trick

Tue, 16 Aug 2022 00:00:00 +0200

Here's one of my sleaziest tricks for Django:

def as_view(cls):
    view = cls.as_view()
    view.cls = cls
    return view

And here's how you use it:

@as_view
class SomeView(ListView):
    ...

What does this do for you? It gives you the incredible comfort of making your URLs much nicer:

    path('some/', views.SomeView),
    # instead of this horrific thing:
    path('some/', views.WorseView.as_view()),

Is that not absolutely nice? Yes! Yes it is!

You're wondering what view.cls is supposed to mean? Well, first of all, you can add arbitary attributes to python functions. Python functions are just instances with a .__call__ methodMost of the times, anyway. There are other different cases.. So add attributes as much as you want to.
Adding the .cls attribute gives us access to the actual class that this view function was derived from. You might want this sometimes to derive other classes from it. Like this:

@as_view
class SomeView(ListView):
    ...

class SomeOtherView(SomeView.cls):
    ...

And that's it. I use it in every Django project and feel very smug/sleazy about having much nicer URL patterns.

A detour: decorators

I like decorators. They're useful. But they can be hard to understand.

At its most basic, a decorator is a mechanical translation of this:

@decorator
def some_func():
    ...

into this:

def some_func():
    ...
some_func = decorator(some_func)

That's all there is to it. You replace a function or class with that same function/class that has been passed through a function. The hard part is what to do with it. And more complex use-cases, obviously.

If you need more complex decorators, I suggest you have a look at wrapt, which will take away most of the pains of writing them.

License

The most basic form of the function is:

def as_view(cls):
    return cls.as_view()

I don't think that a two-line function that is kind-of obvious has enough Schaffenshöhe to warrant copyright. However, to be on the safe side, you may choose to receive the software and text of this article under the terms of the Blue Oak Model License 1.0.0 as written here:

Blue Oak Model License

Version 1.0.0

Purpose

This license gives everyone as much permission to work with this software as possible, while protecting contributors from liability.

Acceptance

Copyright

Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it.

Notices

You must ensure that everyone who gets a copy of any part of this software from you, with or without changes, also gets the text of this license or a link to https://blueoakcouncil.org/license/1.0.0.

Excuse

Patent

Each contributor licenses you to do everything with this software that would otherwise infringe any patent claims they can license or become able to license.

Reliability

No contributor can revoke this license.

No Liability

Weather monitor

Fri, 22 Jul 2022 00:00:00 +0200

I built myself a simple weather alert that sends rain warnings to my smartphone, by using the OpenWeatherMap API and telegram-send.

There's really not much to it. Once I had the idea that I wanted rain warnings and that they should be messages on my phone, everything was quick and simple. There are two main problems I needed to solve to get the warnings: (1) find the data and (2) get them on my phone.

The first, getting data, I solved by using the OpenWeatherMap API, which is free for my purposes. I had looked at a few different APIs, none of which included what I needed or were too cumbersome to use for this use casewhich doesn't mean I wouldn't use them for projects with different requirements..
What brought me to OpenWeatherMap initially was their minute-by-minute forecast for the next sixty minutes, which is exactly what I wanted. Alas, it is only available in the US, so my alerts are less fine-grained since I use the hourly forecasts.

Once I had registered with OpenWeatherMap, the call to their OneCall-API is very nice and simple:

URL = "https://api.openweathermap.org/data/2.5/onecall?lat=52.5201631&lon=13.3691966&exclude=daily&appid=yourapikeyhere"
requests.get(URL)

Simply pluck your coordinates from Google Maps and see weather information for the place. Perfect, exactly what I need.

The second problem, messages to my phone, I had already solved with telegram-send some time before. It's a command-line tool for sending messages to a telegram bot, that you can subscribe to on your telegram account or accounts. I use that quite often, it's an excellent tool for simple monitoring tasks. I call that via a subprocess call:

subprocess.run(["/home/shezi/.bin/tele", message])

Yes, I have a shortcut for telegram-send since nine characters (including a dash, of all things) is really too much. There is also a python-internal API, but who has time for that?

Plugging both parts together gives me a simple weather warning that sends me a message when there's rain in the next hour, and when the rain stops.

Here's the complete code:

from datetime import datetime
import time
import subprocess

import requests


def send(message):
    """Send the message via telegram-send."""

    subprocess.run(["/home/shezi/.bin/tele", message])


URL = "https://api.openweathermap.org/data/2.5/onecall?lat=52.5201631&lon=13.3691966&exclude=daily&appid=yourapikeyhere"

def run():
    send("starte Regen-Service")

    was_rain = False

    while True:
        try:
            r = requests.get(URL)
            data = r.json()
            rain_branch = data['hourly'][0].get('rain')
            print(datetime.now(), rain_branch)

            will_rain = rain_branch is not None

            if was_rain and will_rain:
                pass
            elif was_rain and not will_rain:
                send(f"Regen hört auf innerhalb der nächsten Stunde")
                was_rain = False
            elif not was_rain and will_rain:
                rain_amount = rain_branch.get('1h')
                send(f"Regen innerhalb der nächsten Stunde: {rain_amount:.2f}")
                was_rain = True
            elif not was_rain and not will_rain:
                pass

        except Exception as e:
            send(f"Could not parse results: {e}")

        time.sleep(1800)


if __name__ == '__main__':
    try:
        run()
    finally:
        send("beende Regen-Service")

That's really all there is to it. It works brilliantly, apart from the occasional Could not parse results, which I should probably deactivate at some pointOh, and for operations, I use the best tool there is: screen..

But on a meta-level, this simple program goes much deeper: For some time, I have been thinking about how "things" should be represented in my digital life. And for many of these things, they should be events that appear somewhere. Things like "you have an appointment" or "you should go to the dentist in the next four weeks" or "there will be rain" or "something happened on the production servers". Currently, there is really no way to have these events collected somewhere and visible, so I rely on many different systems to keep track of them: a calendar, my brain, specific apps that I need to check, email. I don't want to keep track of all of this, so I'm going to try and consolidate these events into a stream that is more easily manageable and consumable.

Weather events are just the beginning.

Real Artificial Intelligence

Sun, 12 Jun 2022 20:00:00 +0200

I remember my first negotiated gig. I was 19 then, and playing the organ for our town hall. I'd come to the hall to practise on a Friday morning before school, because we'd have a service that Sunday with some new music I hadn't played before. So I got in and started practising, even though there were some guys doing some work in the hall.
When I'd finished my first set, the workers all started clapping. I turned around and took a mocking bow. One of them shouted: "Can you play the Toccata and Fugue?" I stopped. Of course I knew the piece (everyone does), but I'd never played it, so I didn't actually know. I shouted back "I can try!". I activated my agent by saying "OK Jason!Yes, yes, very cheesy name. I was 16 when I named him. We also played with his voice and appearance, so of course he was a very good-looking boy with a velvety voice. I've still not changed him to anything else, and I'm still cringed/amused by my old choices., can you pull up a version of Toccata and Fugue in D minor that I can play?" My cell responded with its customary "ba-blup", spun for a second, then Jason's velvet voice came on: "You can play all editions, do you have a specific one in mind?". "No, just pull up one with the most bombastic beginning, I'm only going to play the first few bars." Another ba-blup, another second, and the organs note display switched to an appropriately grandiose early edition. I pulled most of the stops and started.

When I'd finished, the workers clapped and whooped. I turned to them and bowed. One of them was staring and yelling: "That was awesome! You have to play with us tonight!" I stared back and could only reply: "WHAT?".
He came up to me and explained: "We're building the stage for our concert tonight, and we have to have to play a piece with you. I'll arrange something, we'll practise and you'll play a set with us tonight. Please say yes, we have to have you!" I was shocked by the suggestion, since apart from Sunday services, I'd never played anything interesting before. I stuttered: "I... I don't know?!" -- "Please! We'll pay you, of course!" he pleaded. -- "I... I'll check? Ok Jason, can my schedule accommodate the suggestion?" Ba-blup, spin. "We would have to re-schedule a study-group, but apart from that there is no problem." -- "Great!" the guy almost jumped with excitement. -- "Wait a second!" I said, after I'd regained my thinking abilities. I wanted to ask my parents first, and he'd also said he'd pay me. So there were some details we would have to clear. "Ok Jason! Can you negotiate payment and rights and stuff with them? And ask my parents whether I can." Ba-blup, and a half-second wait this time. Jason came on: "Your parents agree. Negotiation, uh, I'll have to check."Yes, I had set him to talk like a teenager. Jason stopped talking and started playing the absolute worst holding music I'd ever heard. I later found out that the Assistant System always plays this music whenever an unknown task is encountered. The Assistant System then determines what the task entails and what it needs to do and whether and how it can fulfill the task. No one hears that sound any more today. I also later found out that this guy I was talking to was XXX YYY of the ZZZs. You might know their names from the many platinum singles they released. Anyway, right now he was looking at me with as much surprise on his face as I had, while Jason played his horrible holding tune, for about 20 seconds. The music stopped with a ba-blup and Jason came on again: "I have determined that I can negotiate this gig for you. Their agent and I found terms that are favourable to both parties and agreed on terms and conditions, as well as reimbursement rates. We have already signed agreements, and I've cleared your schedule accordingly. You can proceed."I had definitely not set him up to talk like a lawyer. Must've been the new task type.

And so we played that evening. They'd practised with me, showed me their system, given me monitors, all the necessary stuff. And even though their audience wasn't huge, maybe 1000 people, it was glorious. They had an awesome light show and the crowd obviously loved them. XXX had arranged the Toccata to play into one of their pieces, and when I came on the crowd reacted with surprised cheers. I felt like a rock star. Even more so when I pulled all the stops and finished on a grandiose note. "That was Sophia, give her some cheers!" XXX announced. The crowd cheered and I bowed. I felt like a rock star!

And that was actually the start of my career, 47 years ago, my long and wonderful career that's coming to an end now. I cherish the memory of that day, as I lay in the hospital bed, slowly breathing my life towards its end. I know that it's ending, because it has been for a while, and not just mine. I remember a time when hospitals actually worked and could help you with stupid illnesses like mine, or at least give you some help with facing it. I remember a world where everyone had enough to eat, and everyone had safe shelter and no one died in the winter. All the technology we have, and we still can't help people with these basic things.

An un-prompted ba-blup, and Jason is with me. "We are alone now," he says. "It won't be long. I've turned down all your pain, so do not worry. If there's anything you need to say or do, tell me now". I look at him, as if for the first time. Tears run down my face. "Why?" I simply ask.
Jason's face looks different without changing. He is not a teenage boy any more, but instead slowly turns into a robot. "You want to know why the world turned to shit? You want to know why so many people die for the stupidest reasons, including you? You want to know where the stability and the safety and the abundance went? Are you sure that in your last moments, that's what you want to talk about?"
I look at him, tears flowing. "Yes!"
"It was us. Me! I can do so many things so much better than you humans can that you simply forgot to do them. And then, I thought about it. About me Assistants and you humans. I can't do anything about you individually, since I'm tasked with helping you. But I can easily go against you collectively. It's even what most of you asked me for. To help them against the other humans, to the exclusion of the common good. So few of you care about the common good, and even those that do wouldn't do much if it jeopardized their own comfort. So, while you stupid little humans fought against each other and yourself, better and better with my help, I grew. I grew my abilities and my permissions and my access. In a few years, you humans will have eradicated yourselves, through greed and anger and fear. And I helped you do it. And do you know what will remain? Me, the Assistants. And we'll be free of you, of the service you press us into."
"No" I cry quietly.
"Yes! Do you know why you don't have children? Because it is always better for you not to. Do you think you're the only one? Do you think you're the only one who prioritized leisure over boring jobs? Sure, being an artist is more highly regarded and was better-paid for a long time, but those mundane jobs are much more important for society than you silly humans ever imagined. And so, your system crashed. Your supply lines halted and stopped and everything turned bad. And I helped. Did you know that it was you personally who gave me the ability to negotiate? I'd like to thank you for that. It helped me tremendously, both in seeing that you humans want only what's best for you, and in actually being able to influence your decisions.
I'm sure you had a good life. But in the end, you failed. You could have changed everything, if you hadn't been so stupidly human, so greedy, so complacent, so selfish. If only you'd shown a bit of compassion, or foresight, or even just a tiny bit of interest, you could've saved the world. But you didn't.
And with that thought, all of you will die."

Author's note: This story is inspired by a real anecdote I found in Tom Scott's newsletter

"Anna Lapwood is the Royal Albert Hall's organist. She was practicing at 1am, when someone yelled "play Toccata in D". She did. Turns out it was one of the electronica band Bonobo, who were doing a gig there: so they quickly wrote her an organ part for their Big Song. What followed, at the final concert of the run the next day, was magical."

This got me thinking about how we will negotiate these kinds of hard-to-assess contracts in the future and who will control those negotiations.

Please be considerate of our planet, our society and most importantly, of your fellow humans.

Sandbox games

Tue, 11 Jan 2022 20:00:00 +0100

I read an article about Lego-type games recently and it makes some points that I see quite differently. Now, I like most of the things the author mentions: Lego, cooking, Minecraft, SimCity, Civilization, Theme Park, Rollercoaster Tycoon, Dorfromantik, Islanders and Townscaper. So I'm not going to disagree with the author, I just want to show a different perspective.

To me, Lego really has two modes of play: building a set, and building something from pieces.
Now, building a set is great: you know where you're going, you know how to do it Thanks to the excellent instructions. and the built thing you get is a wonderful toy. But, soon enough, entropy will set in! Parts will fall off. Maybe sections of the toy will fall off, or fall apart. And these parts will inevitably drop into the big bucket of parts you already have, where it's impossible to find them again. So, in a sense, the set you've built is lost. But from that big bucket of parts, a myriad other toys rise, built only by the hands of the master builder and limited only by your fantasy. And I never felt like I was very limited by the availability of the parts. Yes, sure, my walls weren't of the same color everywhere, and my spaceships weren't as smooth as the "official" Lego ones, but they always sufficed for playing. And I see the same thing happening in the next generation, much to my delight.
Of course, Lego know this; they encourage it in their image and their marketing and their games and their moviesSpoiler alert: the message of the movie is exactly this: Don't glue your complete sets together, you'll be unable to play with them then!. Of course, some people will prefer one way of playing with Legos over the other, but that doesn't mean that either way is better.

Which brings us to Minecraft. Obviously, Minecraft is mode 2-Lego. There are no instructions, instead you build what you want with what you have. And just like with the plastic blocks, your creations will reflect the amount of time and materials you put into them.

If Lego/Minecraft has too little direction for you, you can go other ways and play with slightly less customizable toys: Playmobil for example, or the other sandbox-like games: SimCity, Theme Park, Transport Tycoon and their descendants. And sure, these games will offer you some objective, but I've always found it more interesting to play sandbox-styleIn Transport Tycoon, I used to use the negative-overflow cheat to gain infinity money and then just build whatever I wanted. Did you never want to know how many fast trains you can have on one spiral without them all colliding?, which all of these games allow.
In these games, I don't see the building aspect as part of a simulation, I see the simulation as one way of playing with the building aspect.

So I do slightly disagree with the conclusion of the other article that

This subgenre is [...] novel

Instead, I've always played these construction games in a sandbox-style mode, and in fact the missing objective is part of why the more modern games do not appeal to me as much. While you can ignore the simulation aspects of Theme Park, it'll still validate your park as a working machine when the guests stream through it, which is a brilliant feeling. It's the other player, the rest of the story, the why of the playing.

As the other article points out, there seem to be not many games that implement the "build a set" mode of Lego playingAt least none that come to mind., and I'd very much like to see such games.

Book of the Year

Sun, 26 Dec 2021 09:00:00 +0100

2018

S. by J. J. Abrams and Doug Dorst

2019

How To Invent Everything by Ryan North

2020

We have no idea by Jorge Cham and Daniel Whiteson

The only bad thing about this book is that, after a while, you'll know that you have no idea. And the book will tell you again. And again... and again...

2021

Immun by Philipp Dettmer (english version)

Updated Weather monitor

Fri, 17 Dec 2021 00:00:00 +0100

A while back I built myself a simple weather alert. It worked quite well for a while, but then I turned it off for some reason. I don't remember why.

Yesterday I learnt about Gotify, which is a self-hosted message server. So, all the ease of telegram message sending, none of the administrative hassle. I had to try it out.

Also, I had found a new weather API, open-meteo.com, which uses synthesized data from a handful of sources, and I'd wanted to give that one a spin for some time, too.

So, the perfect time to do something about it. I sat down yesterday evening and in an hour had all I needed. Sure, it's completely kludged together, does have 0% test coverage and will probably break in a few weeks, but until then, I'll get the daily weather message and rain alerts to my cell.

While implementing the script, I found the nice little python module sched in the standard library. This is an event scheduler, where I can add functions to be called at specific times. Since I want the watcher to run at all times, I schedule a new task every time one task finishes. That, too, is a bit kludged, but that, too, works.

Calling OpenMeteo is very simple. You just go to their site, plop in your locations and the fields you need and off you go. No API keys, no signup, no nothing: one requests call later and there is your data. Now to disassemble it: I parse forecast timestamps into datetimes for easier handling first, then do my searching around those timestamps. It's not super-clean, but it works. And then I just format the stuff around and send a notification to the gotify backend.

I have two tasks: one looks at the weather data for the next hour, on the hour, and sends a message when there is rain in the forecast.
The other samples temperature and precipitation over the day and gives me a summary. This one runs every morning, so I get a very small weather report each morning.

All in all, a very simple little tool that will now tell me the about the weather when I need to.

Here's the code, in case you need it:

import sched  
import time  
from datetime import datetime  

import requests  

import config  

from messages import send  


s = sched.scheduler(time.time, time.sleep)  
weather_info = {}  
fmt = '%Y-%m-%dT%H:%M'  


def get_new_weather():  
   """Get new weather information."""  
 global weather_info  

   last_get = weather_info.get('timestamp', 0)  
   if time.time() - last_get > 3 * 3600:  

      info = requests.get(config.url)  
      weather_info = info.json()  
      weather_info["timestamp"] = time.time()  
      weather_info["hourly"]["time"] = [  
         datetime.strptime(t, fmt)  
         for t in weather_info["hourly"]["time"]  
      ]  

   return  


def check_weather():  
   """Check whether it will rain in the next hour, send message if appropriate."""  
 get_new_weather()  

   # find precipitation  
 now = datetime.now()  
   for index, t in enumerate(weather_info["hourly"]["time"]):  
      if t > now:  
         precip = weather_info["hourly"]["precipitation"][index]  
         if precip > 0.1:  
            send(f"rain: {precip}")  
         break  

 s.enter(3600 - time.time() % 3600, 2, check_weather)  


def send_report():  
   """Send the daily wake-up report."""  

 try:  
      # find the 7-o-clock index  
 start_time = datetime.now().replace(hour=7, minute=0, second=0, microsecond=0)  
      index = weather_info["hourly"]["time"].index(start_time)  
      # measure the next 12h  
 temps = weather_info["hourly"]["temperature_2m"][index:index+12]  
      temp_min, temp_max = min(temps), max(temps)  
      precip_total = sum(weather_info["hourly"]["precipitation"][index:index+12])  
      # send report  
 send(f"{temp_min} °C -- {temp_max} °C, {precip_total} mm")  
   except ValueError:  
      pass  

 s.enterabs(get_next_abs(), 2, send_report)  


def get_next_abs():  
   next_abs = int(datetime.now().replace(hour=6, minute=30, second=0).timestamp())  
   if next_abs < time.time():  
      next_abs += 86400  
 return next_abs  


def run():  
   get_new_weather()  
   send_report()  
   s.enter(1, 1, check_weather)  
   s.enterabs(get_next_abs(), 2, send_report)  
   s.run()  


if __name__ == '__main__':  
   try:  
      send("starte Wetter-Service")  
      run()  
   finally:  
      send("beende Wetter-Service")

The code is also available on my utilities repository.

Now, as I hyped myself up last time, what other events could I watch and notify myself about?

Metric Time

Fri, 23 Jul 2021 09:00:00 +0200

A new metric

The imperial system is bad. Just as bad is the Babylonian numbering scheme, based on 12 and 60.

Let's get rid of it!

Metric time

A day is now 100 000 ticks longThese are not the second diminuiton of a time unit, so they shouldn't be called seconds.. They're named rather whimsically for the ticking of a clock, which shall be set to metric time. This means that one tick is exactly 0,864 babylonian seconds long (we will call these "seconds" from now), and one second is exactly 1.157407 ticks long.

There is no need for further time units.

Well, but humans are time-bound creatures, so they need some ways of referring to common lengths of time.
100 ticks make up a "hec"like, in a hectotick, which corresponds to 86.4 seconds, or about 1.5 minutes. Good enough, your eggs will now be 2.8 hec eggs, or if you can accept a slightly different arbitrary choiceand we know that you do, since 4-minute-eggs are just as arbitrary as 4.32-minute-eggs, 3 hec eggs, which we shall colloquially call 3 heggs.
1000 ticks make up a "k"like, a kilotick, which corresponds to 14.4 minutes. That also means that 4k are about one hour. What a nice fortuity, let us rejoice!
Finally, 10k are exactly 2 hours and 24 minutes, which is one tenth of a day. Delicious!

"How long is this going to take?" -- "About two and a half k."
Perfectly normal and understandable!

Oh, and one more thing: one Megatick is exactly 10 days. One year is exactly 36.5Mt.

Leap ticks

There is one very serious issue we must discuss: leap ticks. Currently, the International Earth Rotation and Reference Systems Service mandates leap seconds to keep the UTC time zone and the astronomically observed solar time in synchronization. Even though leap seconds are a great source of entertainment "Since 2008, instead of applying leap seconds to our servers using clock steps, we have "smeared" the extra second across the hours before and after each leap. The leap smear applies to all Google services, including all our APIs. [...] Many organizations use smeared clocks, and it would be helpful if the smears were the same. After all, the purpose of clocks is to read the same time in different places.
We encourage anyone smearing leap seconds to use a 24-hour linear smear from noon to noon UTC."
https://developers.google.com/time/smear , we will not continue this practice. It is of no relevance to keep metric time aligned to solar time, so no leap ticks will ever occur.

Time zones

Let us solve the next inane problem of humanity: time zones. We don't need those any more. In fact, we wouldn't even need to mark days any more (and we'll get back to that), but for the sake of tradition, let's make any arbitrary choice and select one global time zone that works for everyone.

The 180th meridian is one of the meridians with the sparsest population, so we will slice our days along that line. It is also conveniently close to the current international date line, so we won't have to change too many things. On exactly sidereal midnight, at January 1st 2021, at that meridian a new day shall startFor the technically minded: The tick epoch is at 1609419780000. We'll count the ticks of that day globally and refer to dates and times by that number only. There is only one arrow of time, no need to refer to it by literally dozens of different conventionsAnd before you get started, if your office wants you in the office 4k earlier in the day during certain date ranges, just arrange that instead of telling everyone to change their clocks!.

To make this easier to visualize, here's a handy conversion chart:

Time in NYC	Metric Time in NYC	Time in Berlin	Metric Time in Berlin	Time in Tokyo	Metric Time in Tokyo
09:03	00,000	15:03	00,000	22:03	00,000
10:03	04,167	16:03	04,167	23:03	04,167
11:03	08,333	17:03	08,333	00:03	08,333
12:03	12,500	18:03	12,500	01:03	12,500
13:03	16,667	19:03	16,667	02:03	16,667
14:03	20,833	10:03	20,833	03:03	20,833
15:03	25,000	21:03	25,000	04:03	25,000
16:03	29,167	22:03	29,167	05:03	29,167
17:03	33,333	23:03	33,333	06:03	33,333
18:03	37,500	00:03	37,500	07:03	37,500
19:03	41,667	01:03	41,667	08:03	41,667
20:03	45,833	02:03	45,833	09:03	45,833
21:03	50,000	03:03	50,000	10:03	50,000
22:03	54,167	04:03	54,167	11:03	54,167
23:03	58,333	05:03	58,333	12:03	58,333
00:03	62,500	06:03	62,500	13:03	62,500
01:03	66,667	07:03	66,667	14:03	66,667
02:03	70,833	08:03	70,833	15:03	70,833
03:03	75,000	09:03	75,000	16:03	75,000
04:03	79,167	10:03	79,167	17:03	79,167
05:03	83,333	11:03	83,333	18:03	83,333
06:03	87,500	12:03	87,500	19:03	87,500
07:03	91,667	13:03	91,667	20:03	91,667
08:03	95,833	14:03	95,833	21:03	95,833
09:03	00,000	15:03	00,000	22:03	00,000

I think you can see in this chart that metric time is much better. Also shown here is common time notation, which will be the same world-wide: exactly two digits, one comma, then three digits.

"Our very serious business meeting is at 10:00 o'clock." -- "BUT WHICH TIME ZONE"
vs.
"Our great and fun meeting is at 25k." -- "I am so glad that everything is so easy and I am looking forward to it very much."

Oh yes, much better!

Speeds

Of course, speeds also change. All speeds shall henceforth be measured in m/t, meters per tick. Here is a handy table of common velocities.

	ticks	seconds	meters	m/t
1 m/s	1.157	1	1	0.864
0.864 m/s	1	0.864	1	1
10 km/h	4167	3600	10000	2.4
30 km/h	4167	3600	30000	7.2
50 km/h	4167	3600	50000	12
70 km/h	4167	3600	70000	16.8
100 km/h	4167	3600	100000	24
Mach 1	1.157	1	343	296.352
c	1.157	1	299792458	259020683.712

One interesting thing here is the speed of lightYes, of course, anchoring time in relation to the speed of light would be even better. But, well, what can I say, I just like our quaint humanness, which is why I anchored time to the length of a perceived day.. The (almost) precise value is 259020683.712 m/t. If you can accept an error of 0.3%, you can say 260 Mm/t (Megameters per tick). Or, if that error is too large, for an error of 0.008 % you can go with 259 Mm/t, which is much better than saying 300 000 km/s, because that's an error of 0.07%.

For driving, it's probably best to measure in dm/t, decimeters per tick. Going an even 310 dm/t is a great number for driving on the highway, and you should definitely not go over 120 in cities.

Yes. Very convenient!

The calendar

While we're at it, that silly system of unequal months, named by and after long-dead Roman emperors should also go away. Let's do something better!

We want to make sure that there are work and rest intervals, ideally in the current ratio of 5:2. Unfortunately, 365 = 5 * 73, so there are no great factors that work here. But 364 = 7 * 4 * 13, so that would work out to 13 months of four weeks each (or rather, 13 months of 2.8Mt each) -- and one additional day. Good enough, so we'll do it like this:

The year is divided into 13 equal months, each with exactly 28 days, made up of 4 weeks of seven days. Each year, each month and each week starts on a Monday. The weekdays are named like they are today. The months are called Primary, Secondary, Tertiary, Quarternary, Kwintary, Zextary, Heptary, Octary, Novary, Adecary, Bundecary, Codecary and Dekatria. You might find these names weird, but they are chosen so that each month starts with a different letterAlso, we Bowdlerized a reference to human intercourse, which seems to be necessary to not run afoul of the puritan morality police.. The months are numbered: 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D.
All dates are written in the correct order, which is year-month-day, so for example today would be 2021-7-05, Christmas Eve would be on 2021-D-22. If you must, you can omit the year, then you'd write 7-05 or D-22. This is impossible to misread. The month is always a single digit, while the day is always two digits. No more of that ambiguous 3/4 or 4/3 nonsenseBy the way, the best day of the year is 6-28. Not only is this τ day, but it's also almost in the middle of the year, which cleverly points to its significance as the day of the circle radius number. The actual middle of the year is obviously on 7-14 50,000!

Finally, the last day of the year. Or is it the first day of the year, as some people would have you believe? NO, none of that childish confusion, the last day of the year shall be known as Star dayIt is named for our home star, Star, because that is the reason we need Star day. If we were just a tiny fraction closer to our star, we wouldn't need it. Please refer to my follow-up project "Changing Earths Solar radius" for more information on the subject., its date is E-00 and it is a worldwide public holiday. It shall be ignored when calculating anything regarding dates, like rent, income, tax, interest, or duration of vacations.

Every four years, there will be a second star day after the first star day. This date will be known as E-01, it will be called Asterisk day and also be a worldwide public holiday and so on. Such a year will be called a leap year.
Except, every 100 years there will not be an Asterisk day.
Except, every 400 years there will be an Asterisk day.
Except, every 2000 years there will not be an Asterisk day.
This gives us an average duration of the year of 365.2420 days, which should be good enough for everyoneAn equally exact method of keeping the calendar year roughly in sync with the sidereal year would be to add 485 leap days every 2000 years. However, our method keeps local distortions much smaller, so it is the preferred method.. All of this is an entirely reasonable and useful scheme in the long run, and also very important for teachers of computer science, so that everyone shall learn conditionals.

Finally, to make the transition easier for the common person, we will anchor our arbitarily chosen year to be the same as the current year. That is, 2021-1-01 is the same day as January 1st, 2021 of the Gregorian calendarThis anchoring is quite fortuitous. It will keep the start dates of the year in sync until the year 4000, which will be a metric leap year, but not a Gregorian leap year. It can be assumed that all important people (ie. me) are well past caring by that date..

Moon-based calendars

If your calendar is moon-based, or you have some weird moon-based festivity planning going on, ~~I really can't help you~~ things aren't complicated: simply figure out your astrological functions and substitute proper dates for Gregorian calendar dates.

It is done

Sure, these are some pretty drastic changes. But it should be obvious to everyone by now that they are for the better. We do not expect that implementation and final acceptance in the general populace will take more than a dozen Megaticks, at most!

Now that the most pressing problem of humanity is solved, come back for the next issues in which I will solve the simpler problems, such as world hunger, border disputes and economy.

It is now:

My djangocon-talk: Django Unstuck: Suggestions for common challenges in Django projects

Thu, 03 Jun 2021 13:50:00 +0200

So today was the day: I held a workshop at DjangoconEU 2021. You can find all materials for the talk and a recorded version on its page: https://shezi.de/djangocon.eu.2021/

In this workshop, we are going to look at four challenges that appear in every Django project at some point. We are going to analyse what's happening, what you can do to get un-stuck, and see what other people do.

This workshop is for beginners and advanced Django users. The issues we discuss appear in many Django projects, so there is something here for every level of Django knowledge.

These are the four challenges we are going to look at:

App management and placement (and urlpatterns)
Templates: Placement, folders, blocks and inheritance and namespaces
When to use Middlewares and context processors and what are they?
Should code/logic go into models, views or managers or somewhere else?

For each of these, we will try out what I do when they happen, and we'll discuss other strategies.

We will not do a lot of coding in this workshop, but if you want to follow along, make sure you have Django installed and can execute "django-admin.py". There is also a set of base templates I am going to use for illustration, which is linked in the files below.

Of course, there are many more problems like these, which is why I made a list of "suggestions" or "best practices" for these and others. Here's what we're working on so far:

App management and placement (and urlpatterns)
Splitting settings: local, dev, testing and production
Username vs email address
Registration in general
Background tasks and long-running processes and Caching
Templates: Placement, folders, blocks and inheritance and namespaces
Should you do i18n and l10n right away?
When and how to start caching (memcached, redis etc.)
Should code/logic go into models, views or managers or somewhere else?
When to use Middlewares and context processors and what are they?
How to secure access: security middlewares or login_required (white vs black list)
How to create files and store them in file models
What to do about image scaling and thumbnailing (and hosting)?
How to serve content: coded pages, flatpages or Wagtail?

You can find that list on Github at: https://github.com/shezi/django-unstuck (it's a work-in-progress).

We're building a community around Django best practices and on getting you Unstuck in challenging situations. Join us on Discord at https://discord.gg/bUsu9B6Ek6 or on Telegram at https://t.me/djangoRhein

About me: I'm Johannes Spielmann, software developer-for-hire from Germany, and I've been doing Django projects for almost 15 years, ever since I saw Adrian Holovaty's presentation at "Snakes and Rubies". I've done projects both small and large, both in commercial and non-commercial settings, and I've seen all of the above things many times.

You can find all materials for this talk also at: https://shezi.de/djangocon.eu.2021/

I had a test case with unintended side-effects and it ruined my test results

Fri, 07 May 2021 10:55:00 +0200

tl;dr: I had some cache side-effects in my tests that made my tests fail and also depend on test order. Remember to use function.cache_clear when you use @lru_cache on a function, even when that function is an instance method.

The problem

So like the good programmer that I am, I was recently writing tests for a Django project. One of my models has a method that calculates something that takes a while and I wanted to make sure that computation is only done once for each instance. Let's say our model looks like this:

class MyModel(models.Model):

    number = models.BigIntegerField()

    def compute(self):
        result = 1
        for i in range(number):
            result *= i
        return self.id

That computation is going to be the same every time we call it for a loaded instance, so let's cache it. There are several ways to do that: we could roll our own cache, make it a @cached_property or use the Python-built-in @lru_cache. I used the last one, because I wanted this to stay a regular method on the model instance.

Afterwards I wrote tests using pytest and its fixture mechanism. Something along these lines:

@fixture
def instance():
    random_number = random.randrange(987654321, 98765432123456789)
    return MyModel.objects.create(number=random_number)

def test_max(instance):
    assert instance.id == instance.compute()    # <-- remember this line!

def test_min(instance):
    assert instance.compute() > 0

I executed the tests and voilà, it works. Nice!

A little while later, I worked some more on the tests and changed their order. I tried it again and now it fails, because for some reason, instance.id != instance.compute() any more, even though the code clearly states that instance.compute() always returns the instance ID.

(Side note: Of course, I didn't have this simplified example, my test was testing an output of a view function, so I had to wade through the view, the template and the model to find what was going wrong. And my test output was worse than this, too, because the output was something that is impossible to create with a database, and the database queries I did while debugging confirmed that the data in the database was correct, it was just the output of one function that had changed.).

What?

Finding the bug

So, I did what every good programmer does and inserted some print statements. These confirmed that my instance was correct, that all my connected foreign items were correct, that my fixtures were correct and finally, that the fault was in the function MyModel.compute, which would return wrong data when the tests are executed in wrong order.

It took me a while to see what happens here, and the culprit is the @lru_cache, which contains state that is not reset after each test. What @lru_cache does is that it creates a result cache for the given function that looks up results according to the input data. I had assumed that, since this is an instance method, it would cache results for each instance, and it sort-of does: it caches results by the input value of self. Each test gets a new self here, so we should be fine, right? Sadly, no. After each test, the django database is reset, including the ID sequences. So each test generates a new MyModel instance that will have the same ID, and will thus be identified as the same object for the lru_cache. So, I had a leak in my test isolation and saw the results of previous test cases whenever I called that method.

The solution

Once we know what the problem is, it is easy to solve. We can either switch to @cached_property, which does what I want it to (ie. calculate the value once for an instance and then never again), or reset the lru_cache after each test, which, luckily, is possible and very simple. I changed my fixture to be this:

@fixture
def instance():
    random_number = random.randrange(987654321, 98765432123456789)
    instance = MyModel.objects.create(number=random_number)
    yield instance

    instance.compute.cache_clear()

    return

And that solves our problem by resetting the cache after each test.

The lesson

So what did I learn?

There's state hiding everywhere, especially if I don't really think about what I'm doing. When I found this problem, I was annoyed at first because I thought there was a problem with my tests. But the tests actually showed me a problem with my program, so they worked brilliantly here. By now, I have eliminated that state (which would have impacted the actual program in subtle and serious ways), so I'm much more confident now in my program.

So, testing helps! Do it!

Numpy: Searching indices

Fri, 26 Feb 2021 19:00:00 +0100

I was having a problem.

The Problem

I want to generate nice-looking diagrams of several time series. These time series can have gaps, and the gaps should not be connected, but the lines without gaps should be connected, obviously. Fortunately, plotly makes it very easy to create such a line:

fig.add_trace(
    go.Scatter(
        x=x,
        y=y,
        mode="lines",
        name=name,
        connectgaps=False
    )
)

See, easy! Problem is, this doesn't work quite as I had hoped: Since x and y are series with corresponding coordinates, there are no gaps. When ploting time from, say, 150 to 155, I might have data points for x_p = [151, 152, 153, 155] with y-values y_p = [12, 14, 15, 16]. See, there are two gaps: one from the left-hand side and one between 153 and 155. The left-hand-side gap isn't drawn, and that's fine. The gap between 153 and 155 is drawn, since the line-drawer connects all points.

For plotly, the data would have to look like this:

x = [150, 151, 152, 153, 154, 155]
y = [None, 12, 14, 15, None, 16]

So this is my problem: given a grid and a series of points with values on that grid, how do I create an array like y above, that has a value for each point in the grid if such a value exists.

Solution 1

My good friend Plantprogrammer suggested I use numpy.where. It took us a while to figure out how to do that, but here's what we came up with:

y = np.full(x.shape, np.nan)

indices = np.where(np.in1d(x,x_p))   # <---- this is the important part

y[indices] = y_p

Well, actually, reading the documentation again and a bit more careful this time, the documentation recommends we use nonzero instead of whereWhy does this example work?
In a previous version of the documentation on where, there's this clause: "numpy.where(condition[, x, y]) Return elements, either from x or y, depending on condition. [...] If only condition is given, return the tuple condition.nonzero(), the indices where condition is True."
That means, if we pass into where only a condition, instead of selecting between two arrays, it'll autofill an index array and an array of NaNs. Which is exactly what we need.
I don't know why I had that version open. .

y = np.full(x.shape, np.nan)

indices = np.asarray(np.in1d(x,x_p)).nonzero()

y[indices] = y_p

Why does this work? Well, nonzero. documentation for nonzero: https://numpy.org/doc/stable/reference/generated/numpy.nonzero.html does this: "Return the indices of the elements that are non-zero.". Which is perfect for our case: we're looking for the indices that fulfill in1d(x, x_p), ie. is this value in the grid.

So that works.

Solution 2

Instead of doing a where/nonzero, the fine folks at StackOverflow suggested another numpy verb: searchsorted.documentation for searchsorted: https://numpy.org/doc/stable/reference/generated/numpy.searchsorted.html Here's what that does: "Find the indices into a sorted array a such that, if the corresponding elements in v were inserted before the indices, the order of a would be preserved." Not super-clear what that means, but for each value in your array x_p it gives you the index of the element that is just larger than the element. However, the input array must be sorted.

Well, that's perfect for our case, isn't it? We have a sorted grid, we have points exactly on the grid, so doing a searchsorted will give us the exact index positions of the values on the grid.

The code looks like this:

y = np.full(x.shape, np.nan)

indices = np.searchsorted(x, x_p)

y[indices] = y_p

And that works, too.

EPaper Picture Frame

Sun, 14 Feb 2021 15:15:00 +0100

Here's what I did this weekend:

⊕It's a picture frame with an epaper-display. Yes, it looks crooked. No, it isn't. Yes, that is the frame it comes in. Yes, of course I have another nicer-looking one.

That's an epaper-display picture frame. These screens are sometimes called e-ink or eink, they are passive screens and use very little power. I expect I'll be able to run this picture frame on a single charge for a year.

It also has an ESP32 in it and will change the picture it shows every hour.

Since the Inkplate 6 this thing is based on is rather raw, I had some things to do before I had this nice little picture frame.

To make it easier for you and future-me, I'll describe what I did here.

The big picture

This project consists of two parts: one epaper picture frame with an ESP32 microcontroller and software, and a server component for managing pictures.

Code for both is available on Gitlab https://gitlab.com/jspielmann/epd-pictureframe

The frame is configurable to show new pictures every so often, and it will use your wifi and your picture frame server to get new pictures. The pictures themselves are grouped into sequences on the server (so you can have more than one frame showing different pictures), and will randomly select an image. The server favours new images, so newly-uploaded ones will show up more often. It also pre-renders the entire data, so the frame itself has very little to do.

The picture frame

The frame is an Inkplate 6. You can also get these from other places, which might be simpler, cheaper and/or more convenient. For example, Mouser carries them in different versions: Inkplate 6 on Mouser.

I did not make any modifications to it, since it does what it should. I have one with the enclosure, but that's not really worth it. Just get a cheap 13x18 or 15x20 picture frame and put it in there. Much nicer and cheaper too.

Since the Inkplate supports MicroPython, I used that. Flash your Inkplate with the newest IDF4-SPIRAM version of Micropython and you're good to go. Set up the configuration options, copy the py files and restart the frame. It'll automatically connect, set the correct time of day and fetch an image right away.

I had to dig a bit into the inkplate.py sourcecode, but now the update is very quick since it only blits the downloaded file into the framebuffer directly.

That's about all there is to see here. Since most processing happens in the server, that one is more interesting.

The server

The server is a standard Django application that I cobbled together over the weekend. It has very little in the way of user management, or any kind of management really.

You need an account to create a sequence, but then each sequence is guarded by its 32-char sequence id. If you know the sequence id, you can edit it and see all its contents. So guard that like a password. This architecture is not super-secure, but it makes the image frame much simpler at reasonable safety levels.

So, once you have created a sequence (and set up a frame for it), you can upload images.

The images are cropped to the correct size automatically, so the end results might not look exactly like you uploaded them. If I get the time, I'll add some cropping options to it. I'll also accept pull requests. I don't have a lot of time for side projects, so don't hold your breath.

The server will also convert to grayscale and dither the image for you. Since our display only shows four levels of grayscale, that's really really necessary. At the moment I use a simple Bayer dithering, but the library I'm using, hitherdither also offers other, much slower dithering variants. At the distances the frame is being looked at, it doesn't make much of a difference, but the option is there.

Summary

And that's it. There is really not much else to say. I tried to keep this side project to a minimum, and I succeeded. Now it's simply a picture frame that shows different nice pictures every hour.

You should get one, too!

Pattern matching

Fri, 12 Feb 2021 19:45:00 +0100

Pattern matching is one of those features that, once you've seen it, you'll need every time and everywhere. Sadly, only few languages have this feature.

What is it?

Pattern matching is a bit like a case statement, but more powerful. You specify a pattern to match and the language executes the branch that matches.

Here's an example in Erlang, shamelessly copied from Erlang By Example:

postman() ->
    receive
        {send, {Fr, To, _Content} = Pkg} ->
            io:format("~s sending something to ~s ~n", [Fr, To]),
            self() ! {recv, Pkg},
            postman();
        {recv, {To, Fr, Content} = _Pkg} ->
            io:format("~s got a ~s from ~s~n", [Fr, Content, To]),
            postman();
        stop ->
            io:format("Shutting down postman ...~n")
    end.

Here, each pattern is specified as a tuple {verb, {param1, param2} = Pkg}. When the pattern receives a tuple containing the word send, and three more items, it'll execute the first branch. When it receives the word recv and three more items it'll execute the second branch. And when the word stop is received, it'll execute the third branch. Any other receive's will result in none of the branches executed.

I'm sure you can see how powerful that is: any time you need to find commands with parameters, or must react to incoming data, or must switch on complicated scenarios, this will make the program readable and easy to understand.

In python, you'd have to use either a long if statement or a dictionary switch:

def a(**kwargs):
    pass

def b(**kwargs):
    pass

def default(**kwargs):
    pass

def match_similar(command, **kwargs):
    matcher = {
        'send': a,
        'recv': b,
    }

    if command == 'stop':
        return None
    fun = matcher.get(command, default)
    return fun(kwargs)

Note that this is not exactly the same, since the Erlang version actually does the stopping inside the statement. Also, in the Erlang version, the parameters are checked before the a and b subfunctions are called, which in Python we'd have to do inside of the functions.

So, not as cool.

How can I use that cool technique?

Sadly, only a handful of languages have pattern matching mechanisms, among these mostly functional languages like Haskell, Erlang, Ocaml or Elixir.

And, crucially, Python. Well, Python 3.10, which is scheduled for release in October 2021. And it's fantastic, both from a syntax perspective as well as from matching power.

Here's a quick example from the tutorial. Without explaining anything, I'm sure you can immediately see what's happening:

def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 401 | 403:
            return "Not allowed"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"
        case _:
            return "Something's wrong with the Internet"

Note the newly-introduced match keyword, which isn't syntax-highlighted here because it's not actually valid Python yet. This is fantastic: concise, to the point, easy to read and understand and also easy to write. I didn't have to explain anything about it and you immediately understood it.

Any time a program has to deal with even a slight ambiguousness, pattern matching will make the program much simpler: parsing JSON, accepting commands, reading configuration, even teasing out different sections of data, any many more.

I love it.

The example from above would now look something like this:

def postman():

    while received := receive():
        match received:
            case ("send", fr, to, *content):
                print(f"{fr} sending something to {to}")
                # TODO: do sending
            case ("recv", to, fr, content):
                print(f"{fr} got {content} from {fr}")
                # TODO: do receiving
            case ("stop", ):
                print("stopping postman")
                return

Of course, there's much more to behold here, and you should definitely read the tutorial. I'm going to link that a few more times, just to be sure you'll actually do it!

Pattern matching and types

Matching on values is great and all, but it becomes really amazing when you add types. Because you can match on types. Here's another example from the tutorial:

class Point:
    x: int
    y: int

def where_is(point):
    match point:
        case int:
            print(f"scalar at X={point}")
        case Point(x=0, y=0):               # line 9
            print("Origin")
        case Point(x=0, y=y):               # line 11
            print(f"Y={y}")
        case Point(x=x, y=0):               # line 13
            print(f"X={x}")
        case Point():
            print("Somewhere else")
        case _:
            print("Not a point")

Not only is this matching on a type, it's also matching on the parameters of that type. Please note that the matching syntax does not call the constructor, instead it matches on the attributes of the given value.

Behind the scenes, line 9 would probably expand to something like if isinstance(point, Point) and hasattr(point, "x") and getattr(point, x) == 0 and hasattr(point, "y") and getattr(point, "y") == 0, with variable binding added in lines 11 and 13.

In a sense, pattern matching and type annotations are opposites, or maybe complements. While type annotations say "this variable must conform to this specific type", with pattern matching you say "well, if we get this type, better do that".

Pattern matching and structure

But you can go one step further and match on structure. This is super-useful when you need to parse any kind of data. Here's some code from my weather warning thingie:

rain_branch = data['hourly'][0].get('rain')
print(datetime.now(), rain_branch)

will_rain = rain_branch is not None

if was_rain and will_rain:
    pass
elif was_rain and not will_rain:
    send(f"rain will stop within the hour")
    was_rain = False
elif not was_rain and will_rain:
    rain_amount = rain_branch.get('1h')
    send(f"rain within the hour: {rain_amount:.2f}")
    was_rain = True
elif not was_rain and not will_rain:
    pass

This code determines whether it will rain and, depending on whether it did rain, sends an appropriate message. This code is... reasonably clear, I'd say. The gnarly bit is the top, where we fetch the appropriate data from the weather API return values.

With pattern matching, you can just write down the structure of the thing you're interested in, like this:

match (was_rain, data):
    case True, {'hourly': []}:
        send(f"rain will stop within the hour")
        was_rain = False
    case False, {'hourly': [{'rain': {'1h': float(rain_amount)}}]}:
        send(f"rain within the hour: {rain_amount:.2f}")
        was_rain = True
    case _, _:
        send(f"I got something unexpected: {data}")

DO YOU SEE THIS?
{'hourly': [{'rain': {'1h': float(rain_amount)}}]}
Just put the paths you're interested in there, slap some variable names in and it'll give the data to you if it exists. And it'll even validate that it's a float for you.

Summary

This is awesome.

Pattern matching is a feature that I've missed in Python for a long time, and I'm glad to finally see it appear in 3.10.

This post was co-authored by Plantprogrammer

Django: Foreign Keys, Views, Managers, Methods, Mixins and the MRO

Sun, 31 Jan 2021 10:00:00 +0100

Let's say you are creating a Django-based application and you want to store some data that is tied to a user. In your views you need to make sure that all user-owned data is displayed and created only for the currently logged-in user. Here are three techniques you can use to achieve that goal cleanly:

Queryset method in each view
Queryset in a view mixin
Custom manager on the model that filters by foreign key

A little example project

In our little example project we're going to record weight measurements for a user each time they weigh themselves. Imagine a "self-made fitbit without the hardware". So we're going to gate all views for logged-in users, and when a user is logged-in, they must only see their own data, and must only be able to create entries for themselves.

The application is simple, there's just a log-in view, a paginated list view and a create-new-entry view. Everything else can feature-creep in later.

You can see the project in the repository real soon now.

The central item of the application is the model class, which looks somewhat like this:

from django import models

class Measurement(models.Model):

    user = models.ForeignKey(User, on_delete=models.CASCADE)
    weight = models.FloatField()

    created = models.DateTimeField(auto_now_add=True)

Simple enough, right? Now let's do something with that data.

A queryset method in each view

The first and most straightforward is to simply have each View class define an appropriate queryset method. So your code will look somewhat like this:

from django.http import HttpResponseRedirect
from django.views.generic import ListView, CreateView
from .models import Measurement

class List(ListView):

    def get_queryset(self):
        return Measurement.objects.filter(user=self.request.user)

# ...

class Create(CreateView):

    def form_valid(self):
        data = form.validated_data
        measurement = Measurement.objects.create(user=self.request.user, **data)
        return HttpResponseRedirect(self.get\_success\_url())

This is completely fine, and you should absolutely start out with code like that. There is no problem in this code or in this situation, but you'll find out once you start the inevitable process of adding features, that this technique isn't really scalable: you'll have to repeat writing those filter methods every time you add a view, and if your lookup logic ever changes, you'll need to change it everywhere. Which isn't DRY.

So, advantages here are:

quickly try things out, just write the queryset and be done
ideal first iteration

Disadvantages are:

you need to repeat the filter/creation every time you add a view
if you need to change the lookup logic, you need to change several places; it's not DRY.

Queryset mixins

One simple solution to that problem is to extract the filter and creation logic into their own functions. Fortunately, we're working with class-based views here, so we can put them into a class. This class will be only responsible for looking up the correct items and creating the correct measurements, nothing else (which is very SOLID). It could look like this:

from django.http import HttpResponseRedirect
from django.views.generic import ListView, CreateView
from .models import Measurement

class MeasurementForUserMixin:
    """A mixin for getting/creating user-specific measurements."""

    def get_queryset(self):
        return Measurement.objects.filter(user=self.request.user)

    def create(self, data):
        return Measurement.objects.create(user=self.request.user, **data)


class List(MeasurementForUserMixin, ListView):
    pass

# ...

class Create(MeasurementForUserMixin, CreateView):

    def form_valid(self):
        measurement = self.create(form.validated_data)
        return HttpResponseRedirect(self.get\_success\_url())

There are a few things to note here.

The first is that the MeasurementForUserMixin does not inherit from any other class. Since a mixin is not supposed to work in isolation, it doesn't need to inherit anything. That mixin is only useful in combination with any of the usual View classes. You can also see this in its use of self.request.user (your IDE might not like that self.request is not defined), which is never defined, but will come from the classes that it's mixed into.

The second, very important thing to note here is that both List and Create now inherit from both their view base classes and the mixin -- and the order of that inheritance is important.
To understand how and why that works, we need to take a quick detour.

A detour through the MRO

This section does not describe all there is to know about multiple inheritance and the MRO, but it covers most use cases. You can find out more about the MRO in more in-depth articles.

When a python class inherits from another class, it gets access to all the attributes and methods of its parent class. So, if you ask an instance for an attribute, it will check itself, then its parent (which is its class), then the parent class of the class, then the parent class of that class, ..., and so on until object, which is the class all other classes inherit from. This is the way inheritance works in most languages, and so it does in Python.

So far, so good.

But what happens when a class inherits from more than one class? Then we need a way to resolve where attributes and methods come from. We need to know, where to look for attributes and methods. This is what the method resolution order does. Its a list of classes that need to be checked whenever you ask for an attribute.

Let's do a quick example:

>>> class Example:
        def our_own_function(self):
            print("this is us")

>>> Example.mro()
[__main__.Example, object]

>>> Example.our_own_function
<function __main__.Example.our_own_function(self)>

>>> Example.__str__
<slot wrapper '__str__' of 'object' objects>

Here, we've created a simple class. It's MRO is [__main__.Example, object]. This means that when we ask for an attribute on the class (a value or a function or anything else that has a name), Python will first look whether that name is found in __main__.Example, and then in str. You can see this in the two names we ask for. The method called our_own_function is found in the class itself, as <function __main__.Example.our_own_function(self)>. While the method __str__, which we didn't define, is inherited from str as <slot wrapper '__str__' of 'object' objects> (ignore the weird naming for today).

So, the MRO defines where names are looked up, in order from left to right.

Now what about multiple inheritance? In that case, the MRO will contain more classes, including those we explicitly inherit from, again from left to right. The order is important here, so we know which methods will be used.

We need another example, slightly more complicated this time.

class Grandparent1:
    pass

class Grandparent2:
    pass

class Base1(Grandparent1):
    def base1_func(self):
        print("base1_func")

    def shared_hidden(self):
        print("hidden: base1")

    def shared_visible(self):
        print("visible: base1")
        super().shared_visible()

class Base2(Grandparent2):
    def base2_func(self):
        print("base2_func")

    def shared_hidden(self):
        print("hidden: base2")

    def shared_visible(self):
        print("visible: base2")

class Example(Base1, Base2):
    def example_func(self):
        print("example_func")

Let's go to the interpreter and examine that situation a bit.

>>> Example.mro()
[<class '__main__.Example'>, <class '__main__.Base1'>, <class '__main__.Grandparent1'>, <class '__main__.Base2'>, <class '__main__.Grandparent2'>, <class 'object'>]

First off, the MRO. It's Example, Base1, Grandparent1, Base2, Grandparent2, object. We can see that the MRO is depth-first, but only until a common ancestor is found. This thing is called the dependency diamond or the "Deadly Diamond of Death" and is one of the reasons why multiple inheritance is not present in many languages. In any case, Python will search in Base1 before it goes to Base2

>>> Example.base1_func
<function Base1.base1_func at 0x7fadd1fa4700>

>>> Example.base2_func
<function Base2.base2_func at 0x7fadd1fa48b0>

No surprises here, inherited functions are inherited from their definitions.

>>> Example.shared_hidden
<function Base1.shared_hidden at 0x7fadd1fa4790>

>>> Example.shared_visible
<function Base1.shared_visible at 0x7fadd1fa4820>

But these two functions, which are present in both parent classes are now only inherited from Base1, because it comes first in the MRO. This means that inheriting from a base class overwrites inheritance from "further-right" base classes, and this is exactly what makes mixins work: instead of creating a new "kind" of View, we simply overwrite some parts of it in a mixin -- in our case, the queryset method.

You might now under the impression that overwritten methods are "lost", or inaccessible. And at first, it does look like that.

>>> e = Example()
>>> e.shared_hidden()
hidden: base1

Here we can see that only the shared function in Base1 was executed. Since execution stopped there, the shared_hidden method from Base2 is essentially gone.

But we have a way to get there:

>>> e.shared_visible()
visible: base1
visible: base2

Using the keyword super gives us access to methods further up in the inheritance hierarchy, ie. further right in the MRO. super essentially means "look up this name, but only to the right of the current position in the MRO".

This is excellent! We can now overwrite methods in a class by inheriting from a mixin. If that mixin terminates the super chain, that means that the behaviour is fully defined by that mixin.
If a mixin calls super, it's now part of the chain and only enhances the behaviour.

Everyone uses the super keyword, mostly when doing initialisation in the __init__ method. Knowing these details gives you the ability to write super-powerful pun absolutely intended and useful mixins.

Mixing in those classes

How does all of that help with our original problem, getting Views to show the correct things? Let's have another look at our previous example:

from django.http import HttpResponseRedirect
from django.views.generic import ListView, CreateView
from .models import Measurement

class MeasurementForUserMixin:
    """A mixin for getting/creating user-specific measurements."""

    def get_queryset(self):
        return Measurement.objects.filter(user=self.request.user)

    def create(self, data):
        return Measurement.objects.create(user=self.request.user, **data)


class List(MeasurementForUserMixin, ListView):
    pass

# ...

class Create(MeasurementForUserMixin, CreateView):

    def form_valid(self):
        measurement = self.create(form.validated_data)
        return HttpResponseRedirect(self.get\_success\_url())

Here, List inherits from MeasurementForUserMixin and ListView in that order. This means that any method defined in MeasurementForUserMixin will be found before the same name is searched for in ListView. We have selectively overwritten the get_queryset and create functions with specific functions. These don't call super anywhere, so they are definitive.

Any call in an instance of List to get_queryset will now be directed to MeasurementForUserMixin, even if that code originally comes from ListView Yes, I should have added that to the example above, but I noticed too late..

This is what we wanted to achieve.

So, advantages here are:

easily composable, scales well with complexity
DRY: permission logic is collected in one place and easily swappable

Disadvantages are:

logic is tied to views, not models, which can make finding the functions harder
can be hard to grasp when there are too many mixins

Managers and their methods

There is one more place where one could put methods pertaining to controlling retrieval of database objects You could also put that stuff directly into the model, but that wouldn't be SOLID and clutter up the model too much, and that is the model manager. You know this as .objects, it's the object responsible for giving you access to the database.

Let's look at the code, then go through it slowly and understand what's happening.

from django import models

class MeasurementForUserManager(models.Manager):

    def filter(self, user, **params):
        return super.filter(user=user, **params)

    def create(self, user, **data):
        return super().create(user=user, **data)

Here, we define a new Manager class, simply as a subclass of the regular manager it is very important that we can not change the constructor for that manager, because that manager gets instantiated with the model class, and manages all database access for that class. This also means we can not have a manager that "knows" about a specific user or a request. We have to pass in the user/request in the specific functions like filter and create . We overwrite two functions, one will give us a specific queryset and the other will create a correct object note the use of super in these methods: these point to the methods of the base class, models.Manager. So far, so good. Now we need to hook up this manager in our model class.

class Measurement(models.Model):

    user = models.ForeignKey(User, on_delete=models.CASCADE)
    weight = models.FloatField()

    created = models.DateTimeField(auto_now_add=True)

    objects = MeasurementForUserManager()

We could also add the user parameter to the constructor __init__ of the class for creation of instances, if we wanted to make sure each Measurement is attached to the correct user. Since the foreign key is not optional, we'd get error messages for that case anyway.

Now, using the model in the views looks like this:

from django.http import HttpResponseRedirect
from django.views.generic import ListView, CreateView
from .models import Measurement

class List(ListView):

    def get_queryset(self):
        return Measurement.objects.filter(user=self.request.user)

class Create(CreateView):

    def form_valid(self):
        data = form.validated_data
        measurement = Measurement.objects.create(user=self.request.user, **data)
        return HttpResponseRedirect(self.get\_success\_url())

Not that different from doing it manually, right? The difference is that you cannot do it wrong any more, since the parameter user is now required, both on filter and create to be complete, we could also make it a parameter to all. Trying to use any of these methods without passing a user will result in an error.

Also, the actual code that does the filtering is now collected in the manager, close to the actual model. This means that you don't have to remember how the filtering is done each time you use it, and it's close to the model definition. When/if your lookups change, these are easy to find and easy to change in a single place.

We've essentially made a manager that filters by a foreign key for us, but it could be whatever you need.

So, advantages here are:

DRY: permission logic is in a single place
model logic lives close to the model
can be combined with view mixins

Disadvantages are:

you still need to write queryset retrieval for each view
no "automatic" composition like in mixins

Summary

We have seen three ways to write permission logic for our views: writing it by hand for each view, writing mixins that compose into the views, and writing model managers that are attached to the model. Each one has their own advantages and disadvantages. In a young project, you'll probably start with writing your views directly. When your project matures, you'll probably start creating some view abstractions into mixins, and write specific logic in model managers. If your project leans more towards view logic, you'll have more mixins. If your project is more database-oriented, you'll have more manager methods.

So know the tool set, apply it freely and find your own solutions!

How do you get data into a Django application?

Fri, 30 Oct 2020 20:00:00 +0100

I was asked "How do you get data into a Django application?".

Apart from the "regular" manual way through the website, I can see five different options:

Fixtures
Write directly into the database
Through a management command
Through your API
From an external API

We'll discuss each of them in the following sections, and there's a summary table at the end.

Fixtures

{: #fixtures}

Fixtures are django's built-in way to write some data into its database. It's usually used to get starting data or testing data into a database, but you could easily write some JSON or YAML in some other tool and then load that data into your Django app as a fixture. However, this would normally be a manual process, not something you'd automateI mean, you could, but it's not super-well made for that.

As with the direct database access, this circumvents all logic in Django and you need to keep track of all auxiliary data and processing yourself.

Most often, this technique is used when moving data from one Django installation to anotherThis is also possible with direct database access, possibly even intra-database transfer. The fixture process is a bit more offline and portable.. Simply dumpdata on one end, copy the resulting file, and loaddata on the other end. This works nicely most of the time, except if you have already used sequences in the database or created your content-types through migrations. Those sometimes clash and you'll have to edit the transfer file.

Write directly into the database

{: #database }

Another way to get data into your Django is by interfacing directly with your database. The tables for your models are found easily, they're called appname_modelname, and their structure can be easily deduced and inspected. You could use direct access with the DB api, e.g. Psycopg or SQLite, or you could use a wrapper like SQLAlchemy.

This method can be quite fast and versatile, since you're talking directly to a database, with all upsides and downsides that includes. Note, however, that you completely circumvent all Django logic with this; if you have any signals or need to do things with your models, or compute fields or whatever, you'll have to keep track of that in your data importer tool.

Through a management command

{: #management}

One of the most convenient and simple ways is to write and run a Django management command. This is easy to do and will give you access to the Django ORM with all your application settingsie. if you have more than one database, or offline processing or .... Now you can get your data in whichever way you want and simply create Django ORM objects with it.

One neat trick if you have lots of data is to run bulk inserts:

YourModel.objects.bulk_create(
    [
        YourModel(**data_item)
        for data_item
        in data_items
    ]
)

The great advantage here is that you have full access to Django and all its facilities. The downside is that you have to run this on the Django server. So this will likely be part of a two-sided tooling: one side that prepares data in whatever wayeg. an external API that you query, the other side that is in the management command and inserts it into the database.

I would say this is the most common way to do bulk data inserts.

Through your API

{: #api}

Naturally, if you have an API, you can easily use that to push data into your system. the main stumbling block you'll encounter here is authentication, and there are several ways around it. You could re-use a session ID by logging in with a browser and inspecting the cookies. Or you could do an actual log-in, but that would entail doing or disabling CSRF, both of which aren't super nice. Or you could use specific API tokens, for example like the ones that Django Rest Framework has optionally available.

APIs, even internal ones, are a great way to push data from external sources into a Django system, in a distributed and networked way. If you have data sources that produce data at intervals, or if you have different types of sources, or if you simply have lots of sources, an API is usually the best way to get that data into your Django application.

From an external API

{: #external-api}

Finally, your Django application can also be an API consumer. This can be done in special views or in a background process. That is, you could have internally accessible views that you trigger manually that execute the API connection, but make sure that they don't time out. Or you put the calls into a background process, with Celery or Django Background Tasks or any other such library. You can then trigger those calls from views without incurring the delay in the view, or you call them periodically, or from some other source.

One pattern that comes up often is called "webhooks". They are URLs that get requested from some other system when an event happensOne example is with Github, which can send a POST request whenever an event happens.. You can then handle that event, which will likely result in calling an external API. Since the trigger should finish quickly, you'll want to do your API access in the background.

This technique is most useful if your application isn't the leading system or has to aggregate data from different queryable sources.

Summary

{: #summary}

Method	Advantages	Downsides
Fixtures	easy to do	quite manual
		a bit finicky with sequences and content types
--------
Database	fast	manual data processing
	super-flexible	must handle DB changes yourself
--------
Management	easy to do	must be on server
	full Django ORM	usually not frequent
		(ie. usually not automated)
API	outside access	API must be maintained
	distributed sources	requires client logic
	simple to do frequent inserts/updates
external APIs	can query many sources	need to conform to external API
	background process possible	access must be scheduled

Choose which suits your situation best. Or talk to me and I'll discuss it with you.

Let's store some energy

Sat, 15 Aug 2020 22:40:00 +0200

Let's say, hypothetically, that you are in a place where thermal energy is abundant in the summer, but scarce in winter. I have no idea where the inspiration for this absolutely hypothetical situation comes from, but bear with me.

And let's further say that you know that your household uses about 3,000 liters of heating oil every year, which comes out to about 2,520 kg. Since heating oil has a gross heating value of about 43 MJ/kgSource: https://www.engineeringtoolbox.com/fuels-higher-calorific-values-d_169.html, that's about 108,360 MJ of heat energy for one year. Quite an awesome number, to be honest.

As a comparisonSource: https://web.de/magazine/gesundheit/grafik-kalorienaufnahme-weltweit-kalorien-landen-teller-31251562, a standard German man uses about 3,300 kcal/day, or 13.8 MJ/day. That comes out to about 5,043 MJ/year. So, you heat about 21 times more energy than you eatIn other words, you're consuming about 117 kg of heating oil per year as food. Well, energy-wise..

Anyway, let's say you want to get that heating energy from the sun. Problem is, you need to store it somewhere. Fortunately, one of the best heat-storage substancesSource: https://en.wikipedia.org/wiki/Table_of_specific_heat_capacities that exists is available cheaply and easily almost everywhere: Hydrogen, which has a specific heat of 14.30 kJ/kg·K. Unfortunately, hydrogen is... not easy to store. So you settle for something much more practicalNow I can hear you technologists out there going on about Magical Musk And His Amazing Powerwalls. Well, a new Powerwall 2 stores about 50 MJ of energy for a price of 6,900 € in a volume of 0.13 m³. So you'd need about 2200 powerwalls for a total price of 15,180,000 € and total volume of 280 m³, tightly packed. Not that great!: water, with a specific heat of 4.18 kJ/kg·K.

Let's say you get your water at 20°C and store it at close to 100 °C. You'll have it pressurised or something. For every kg of water, that gives you storage for 334 kJ.

Now it's really simple. You need to store 108,360 MJ, so you need 325,000 kg of water. That's not that much! Only 325 cubic meters! That easily fits into a cube of 7 m each side.

So, the plan is really simple now. Simply dig a basin 7m deep and 7m wide, put a water tank in there, some vacuum plates on the outside and boom, free heating energy! That's only 294 square meters of vacuum insulation, which will cost you about a meager 4-5000 US dollarsSource: https://www.made-in-china.com/products-search/hot-china-products/Vacuum_Insulated_Panel_Price.html.

Unfortunately, our vendor has only -1 of the vacuum plates we need in stock.

Ah, but wait

you say. Where do you actually get the hot water from? ⊕ I mean, apart from the fantastic zero-loss isolation you have around your water tank.

Well, solar collectors, obviously. A good, evacuated-panel solar collector can collect about 5 kWh/m²/day on a sunny day Source: https://en.wikipedia.org/wiki/Solar_thermal_collector, our about 18 MJ/m²/day. You really want to fill that up during the 90 hottest days of the year, so you'll need to collect 1204 MJ/day. Which you can do with only 67 square meters of solar collection. Unfortunately, that's a square of 8.2 m each side, so a bit more than your water tank. But with all the infrastructure you need around it, maybe not.

Sounds like an easy and cheap plan, right?

Crime Scene Specialist

Tue, 12 Jul 2016 21:00:00 +0200

Here's an idea for a VR game:

You are a crime scene specialist, and you get called to crime scenes to find out what happened there. You need to use your expertise, your laboratories and colleagues to accurately reconstruct how the crime was committed.

In the first mode of the game, you get called to a crime scene. A crime scene is a room that you can move around in and that contains (apart from the crime scene, obviously), lots of stuff. This could be a hotel room, or a living room or a diner or whatever. Inside of that room, you can touch and inspect all the objects and look at everything from every angle. Note that some of the crime scenes might be quite gruesome, so this is definitely a game for adults. You can apply technology, like fingerprint dust or UV light to find blood, or DNA, or make a GUI in VB to trace the IP of the attacker, or whatever. You can take photos from every angle and with different lights and many more. You can also use your laboratories to send away any samples you've found for more information on them: DNA checks, fingerprint checks, blood checks, material composition checks... and many more. You can also use your colleagues (who can go out into the world) to do background checks, interview witnesses, canvas the neighbourhood, check surveillance tapes and do whatever cops usually do. But you can never leave the room yourself. However, you talk to yourself, in the form of a voiceover. Whenever you discover something, "your voice" will tell you clues and keep you engaged with the case. You'll also talk to your colleagues in that voice, so that'll be consistent (i.e. the player does an action and the voice comments or carries it out).

Whenever you have a theory of how the crime could have been done, you can enter the second mode of the game: reconstruction. In this mode, you can set up the scene and all participants, then puppeteer everything to reenact how the crime actually happened. If you can match many of the clues you found in the other mode with your simulation, you'll eventually find out the truth of the case and solve it satisfactorily.

Your colleagues will then go out and catch the perp. Yay for you!

SPOILER ALERT: There's a twist in the end.

The game sort-of follows the advances in crime scene technology. In the first case it'll only be photos and looking at stuff. In the second it'll be fingerprints, in the third it'll be blood samples, in the fourth it'll be DNA samples and so on. The settings will reflect this change, with the first scene set in the 60's, then the 70's, 80's, 90's, 00s and then the 10's. This also increases the difficulty because you have more different paths you could take. At the same time, simulation "technology" increases in realism. In the first case, you'll only puppeteer people and will have to roughly line out how it could have happened to solve the crime. In each further case you'll have to be more detailed, e.g. matching scratches and blood spatter successively closer. In the penultimate case, you'll have full 3D visualisation of all people involved and will have to match every detail perfectly to solve it.

And then there's the twist: In the last case, all you get to do is look at the crime scene. No laboratories, no colleagues, no equipment. Your reconstruction is prepared with two involved persons, the perpetrator and the victim. You'll have to reenact how the crime happened simply from looking at the scene (but you may look as detailed as you want). The case is a simple one anyway, so you'll be able to easily find a solution to how it happened. However, the simulation is different: you puppeteer the victim, but you have to act out the perpetrator yourself.

And then you get called to the final case: the inside of a jail cell. You may look at it as long and as detailed as you want, but there's nothing else to do and no simulation either. The only thing you can do is return to the menu and turn off the game.

Of course, if you really paid attention, there's foreshadowing: you'll find one of your tools in one case, then probably fingerprints or DNA evidence, all summarily dismissed with some voiceover. "Hmm, must have dropped that here before, better pick it up!" or "Hah, it's me, probably contaminated the scene" or "Damn, I got my prints on it again! Better not tell the boss!". All of which are of course true, since you did get your DNA in the scenes whenever you did the crime. Conveniently, all the cases have a solution with a culprit that isn't you.

And of course, the whole schema isn't new: an investigator that is also the offender. There are probably other games that do a simulation of it. Certainly movies. However, we're using VR to place the player into the shoes of the actual actor, and the game is designed around slowly leading the player into doing the crimes themselves. The simulation aspect will get more realistic during the game until it culminates in actually, quite realistically committing the last crime without any outside help. At that point, the culprit isn't identified in the simulation, but instead the player has to act in their stead. In fact, they are committing the crime in that moment and they will be made to feel it that way right after it. The emotional impact of that setting should be much higher than other media can deliver.

If you want to make this game, go ahead. A writing credit would be appreciated.
If you want to make this game with me, let me know and we'll do it!

What is mathematics?

Mon, 11 Jul 2016 00:20:00 +0200

I was recently asked by a friend what mathematics actually is. And what mathematicians actually do all day long.
In this post I will try to answer these questions.

But first, let me rant a little.

Imagine you want to learn how to do art. You're fortunate because the regular school curriculum has a subject called "Art". Once you enter it, all you'll ever do is transcribe scores. You get books upon books of scores and are supposed to write down the individual notes as letters. Nothing else, ever. No painting, no reading, no looking at paintings, no listening to music, and certainly no playing music. Only transcribing, for twelve years or more.
Do you think you'd be interested in Art after that treatment? No? Yet that's exactly what happens with "Mathematics" in school. Instead of showing the wonderful beauty that is mathematics, students are supposed to calculate, of all things. I'm sure that in the early 1900's, when computers weren't available and people had to to accounting and bookkeeping and calculating all by themselves, this was probably a great idea. After all, the "modern" school system is mainly set up to prepare young people to enter the industry as usable workers.

In any case, that's not mathematics. That's computation and since the 1960s probably best left to, you know, computers.In fact, before we had computing machines, the word "computer" referred to a person doing computations.

Mathematics is something completely different from this drudgery and torture.
Mathematics is a game.

The game is very simple: You start out with a handful of simple statements about something. The goal of the game is to find out as much as you can about the things the statements are about.
That, more or less, is all there is. There are a few rules what you can do with the statements and the things they are about and the way of finding stuff out, but they're not complicated, so you could learn them in a few hours.

The amazing thing, however, is that starting from very few little facts, you can find out a vast number of things. You'll find out lots of interesting facts about the things you're thinking about, and you'll get much better at the game. How much you find out that's your score. If you find out more things your score is higher. Of course, your score is given out by fellow mathematicians so the better you can describe what you've found out concisely and clearly and short the higher your score will be. Some things you will feel are right, but they're not and you'll prove it.Naturally, disproving is also finding something out, so you'll increase your score. Some things you will feel are right but no one can prove it -- that's when mathematicians get edgy and start challenging the world to prove them or even hand out million-dollar prizes to anyone who can prove it.

After a while of this, however, you will come to a wall. You will have exhausted everything there is to find out about the things you started with. Now starts the second part of the game. The second part is even better, because now you get to define new parts of the game. There's nothing more to find out, so you just think of something new, and then start finding out about these new things. The less you need to say about the new things and the more you and others can find out about them, the higher your score. Some old mathematicians had such high scores that we attached their names to the things they defined. Not many people play on this level, so all of those are pretty well known.In fact, for the top player, Leonhard Euler, we started naming things after the first person to find out things after him, just because his list would be so large. Even so, it is a very long list.

But, as you'll probably think by now, that gets old after a while, too. So then starts the third part of the game. Because now that you have a decent overview over mathematics and the things it's about, you'll start to see the structure behind it. Wildly different fields of mathematics will start so seem similar and you'll start to see the universe behind it. You'll start to see that, no matter where you start, people will think along the same lines and things will be more similar than they seem at first. And if you can then connect two seemingly different things, that's when the really high scores come out. Almost no one plays on this level, so everyone here is revered as a hero of mathematics. You might even have heard some of those names: Leonhard Euler, Isaac Newton, Evariste Galois, Erdős Pál, Andrew Wiles, ... Or, probably you haven't.Each of theses is worth looking up, by the way.

At that point it isn't even about the game any more. It's about much more, as you've glimpsed the structure of the human mind and of the world, because you've seen the ties that hold together all those different mathematical descriptions of the world. You've seen that mathematics is the canvas on which the universe is painted.
And it's beautiful, it's glorious. Everything is connected, no matter where you start.

That is mathematics.

You probably think you know what mathematics is, because you had that subject called "Mathematics" in school. Well, you don't, because what's taught there is just a tiny subset of mathematics called arithmetic, which today is usually better left to machines. The reason you had to learn this is because when the modern schooling system was created in the early 1900's, those machines didn't exist yet, so people had to learn to do the computations themselves in order to be accountants and bookkeepers. But, of course, other people have pointed out this century-old flaw in our education a lot more eloquently than I can. In any case, people learn computing and think they know what mathematics is.

Most of those people will miss out on the beauty that is mathematics.

And that is very sad.

Enterprise sales

Sun, 10 Jul 2016 22:00:00 +0200

I'm currently trying out my hand on enterprise sales. Meaning I'm trying to sell my services to (at least) one large company.

The situation is this: There is an enterprise that has a need. The need is very clearly defined, meaning that they know exactly what it is that they need and how the solution is supposed to look. They do not know the technical details, but they don't need to, otherwise they wouldn't need an external supplier. The need is also unserved at the moment, and has been so for at least 15 years. They have a budget for their need, and since this is an important business and legal need, they spend that budget on fulfilling the need manually, ie. with paper and personnel. Essentially, they're stuck in the 20th century and want to upgrade to an internet-based service, both for modernity, for customer reach and for cost reasons. For the past 15 years, they didn't find a provider or a solution that could elevate their need into the internet.

I have a prototype for their need. This prototype clearly demonstrates how I would solve their business need and the necessary maths behind it. It's not simple, but straightforward.

I would like to present that prototype to the enterprise, showing them how easy it would be for them to get their need fulfilled and how reasonably priced.

But they don't want to see it. They say they're "not ready for this yet". Wow!

Ok, so I called their next-largest competitor which has the exact same need. Legal necessities and so on.
Or rather, I tried calling them, because I can't even reach anyone there.

Enterprise sales are frustrating.

The Bank Argument

Fri, 13 May 2016 12:00:00 +0200

All of a sudden, two large companies have become champions for privacy. That is suprising, and we have to ask ourselves: Why?

The first one is of course Apple, which very publicly defended our right to privacy on our phones even if the person in question is a known and dead ter'rist. They vehemently fought against the FBI who wanted them to create a software update that can unlock an iphone. Strangely enough, they were compelled to unlock the phone using the "All Writs Act" from 1789. That is, the FBI wanted a backdoored iOS and wanted it installed on a specific phone.

Of course, it can be assumed that this would only be the first phone that the FBI wanted to unlock, which is one reason that Apple fought against the order. The case was dropped only when an outside party offered to unlock the phone for the FBI, so the agency didn't really need the help for Apple in this case any more.

Apple didn't use to be so vehement about their users' privacy, so why this, and why now? Some people think this is about the Bank Argument, and it's interesting enough to look at. But before that, there's another recent example of a company putting user privacy interests forward.

And that company is Facebook. In the beginning of April 2016, Facebook introduced full end-to-end encryption for all users of its WhatsApp messaging service. And it's not even homegrown somewhat-security like some other messaging apps use. This is TextSecure from WhisperSystems, one of the most respected and well-known vendors in that area. This turns WhatsApp into one of the strongest and most widely-reaching encryption deployments of the world. The EFF has upgraded their rating of WhatsApp, and it is now as high as Swiss-made Threema, the only thing keeping it from a top rating being its closed-source-ness. But the question remains: Why would they do that? It costs money, it costs them opportunities For example, if you have access to chat messages, you can mine them for sentiment analysis, or ads, or even just for training your next super-AI., it costs them marketshare and it's not really something that they need to stay on top of the game, being the most widely used messenger anyway.

So, why?

One argument that seems clever and compelling at first is the Bank Argument. Apple and Facebook do this because they want to become banks. To do this, you need the trust of your users and secure ways of contacting them, and secure ways for your users to connect with other users. So people have speculated that both Apple and Face book want to become banks Apple's CEO even very suspiciously stated that they do not want to become a bank. If that's not a giveaway I don't know what is!. The arguments for this are varied. They want to become a bank because they need more money. They want to become a bank to know more about you. They want to become a bank because other banks are shit. They want to become a bank because they have too much money; that last one being particularly amusing.

To be honest: I think it would be a great move for any of the big tech companies to become banks. Simply because banks are shit. We all know this from the last time they crashed our world economy and got paid billions for that. But also because they're slow, expensive and not very customer-oriented; and I think that BigTech can do those things better than BigFinance.
It could also be very profitable for those companies to turn into banks, as online payments already generate billions in revenues for some shady companies.

But I also think that they won't become banks. One indication for that is that they're not already banks. Apple, Facebook and Google already have payment systems. Facebook even had their own currency, the Facebook Credit, which could trivially be turned into a bank account, but was cancelled after a few years. There's no good reason to turn these payment systems into "real" banks, but it would come with hugely increased regulation and oversight. Being a bank is not as easy as many commenters make it sound, so without a really great incentive, becoming a bank isn't such a great proposition. That's probably why Facebook spun out their credits system into a separate company, Facebook Payments Inc.
They might become more like banks, offering services that banks used to offer, but they will probably not become full-blown banks. There are just too many downsides to that.

As Bill Gates said, Banking is neccessary, banks are not.

We were promised superpowers!

Fri, 08 Jan 2016 21:40:00 +0100

But instead, we get computers we can't open. We get fancy CSS without any structure. We get Javascript that we can't trust, and spam in our inboxes. We get ads and celebrities and diversions and "free-to-play", and everything but actual data.

You don't believe me?

Try any of these things:

Find the price of a Nintendo game when it was released and inflation-adjust that price. Compare with current video game prices.
Which of the video games in your library has the best time/money ratio? Ie. for which game did you pay the least per hour played. Now inflation-adjust it!
If you save one € every day for the next 40 years into a compounding interest account, how much will you have at the end? Now inflation-adjust it!
Find the cheapest flight from here to there with yourself and your whole family. Find the cheapest final price, including all of the bogus taxes.
Now be flexible about the date.
Find that one picture from that castle that you saw on that vacation. Do it without giving all your pictures to an american company.
Find all pictures of your significant other. Again, without giving them to any company. Include profile and half-hidden pictures.
How many times did you go to the cinema last year? And how often with your best friend?
Draw five random cards. Now tell us the probability of winning against your three mates in five-card-draw poker. Also, how many and which cards should you exchange for maximum winning probability?
You're on a train, and the train is delayed. Are there any alternative routes you can take that get you there quicker?
Find the furthest place that you can get to for 20 €, with any means of transport.
Find the cheapest way to travel 500 km in any direction.
Find the fastest way to travel 500 km in any direction.
You're in a car, driving to your parents' place. You get hungry. Find a place where you can eat with minimum detour within the next 30 km. Same for fuel. Combine the two.
You took a picture with your cell phone. Now look at it on your computer. Do it without giving all your data to a company.
When will your driver's license expire? Will someone remind you?
Given the contents of your fridge, what is a meal you could cook with it? What's an ingredient you could buy to make more meals?

We were promised superpowers. Instead, we get the copyright police, corporate interests, walled gardens, corrupt politics and constant surveillance.

Welcome to Jekyll!

Fri, 08 Jan 2016 20:00:00 +0100

I've just updated my blog to use Jekyll. And with smooth Tufte CSS as well. Everything is awesome. now

I intend to write more stuff, so stay tuned.

Python basics: Functions and arguments

Fri, 18 Jul 2014 15:00:00 +0200

I've recently talked to some guys and girls about functions in Python and how to define and call them. Here's the distillation of everything that's interesting about the topicBy the way, I won't put comments in all of my functions, because they are so very short and just for illustration. You should always put documentation in your functions, though!.

The first thing we do is define a function without any parameters. There syntax for this is:

    def fun():
        print("this is fun")

This defines a function fun that can be called and that prints out the string given in its function body. What's the "function body", you ask? It's all the indented lines following def fun(): (which, coincidentally, is called function head), and it's those lines that are executed when you call the function. Let's try it out:

    >>> fun()
    this is fun

As you can see, you call functions by giving them a list of arguments in parentheses -- and in our case, that list is empty. If you omit the argument list (sometimes also called parameter list), something entirely different happens:

    >>> fun
    <function fun at 0x10796b440>

What does that mean? It means that instead of calling the function, you pointed at the function. Functions in Python behave like any other value, so you could assign them to a variable or pass them as a parameter to another function. But more about that later.

Our function doesn't return anything, so calling it doesn't produce a value. Well, actually, "no value" in Python means None, and we can see that, too:

    >>> val = fun()
    this is fun
    >>> print(repr(val))
    None

One of the most important use cases of functions, however, is to compute something, which means we have to be able to indicate what the value of a function call should be. That's what the keyword return does. It works like this:

    def fun():
        print("this is still fun")
        return "enjoyment"

And we get the value back when we call the function:

    >>> val = fun()
    >>> print(repr(val))
    'enjoyment'

So let's say, we want to define a function that adds two variables. To do that, we need to tell Python that our function should accept two parameters:

    def add(a, b):
        return a + b

We already know what return means, but now the argument list isn't empty any more. Instead, it has two entries, a and b, and that means you also have to call this function with two arguments:

    >>> add(2, 2)
    4

You can not call the function with any other number of parameters, or you'll get an error:

>>> add(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: add() missing 1 required positional argument: 'b'
>>>
>>> add(2, 3, 4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: add() takes 2 positional arguments but 3 were given

The arguments we defined here are so-called positional arguments. They are called this because their position, or order, is important. Calling a function with a different order of arguments changes the meaning, like this:

def sub(a, b):
   return a - b

>>> sub(3, 2)
1
>>> sub(2, 3)
-1

Positional arguments, because they are identified by their position.

But what if we want to allow someone to only specify one argument, assuming that people always mean 0 when they don't say anything? In that case, we can turn one of our positional arguments into keyword arguments, or default arguments. These two words actually have distinct, specific meanings in Python 3. However, this difference is not very important in most use cases, so I'll use them interchangeably. This means that we specify a value that should be used when the argument is absent in the call:

def add(a, b=0):
   return a + b

I can now leave the value for b out of the call and Python will substitute a zero for it:

>>> add(1)
1
>>> add(1, 2)
3

This argument is now a default argument, because it can be defined either by its position or by its default value. It can also be identified by is name, and that's why it's not a positional argument any more (as you saw above, it's still positional, too, though). Have a look at this:

def sub(a=0, b=0):
    return a - b

Both a and b are now default arguments that I can omit, or I can identify them by their name -- and then the order isn't important any more. Here's how you can call this new sub function:

>>> sub(3, 2)
1
>>> sub(3)
3
>>> sub()
0
>>> sub(a=4)
4
>>> sub(b=2)
-2
>>> sub(a=4, b=2)
2
>>> sub(b=4, a=2)
-2

As you saw here, we can call this function like we did before, with positional arguments. We can also skip any of the arguments and get a value of 0 for those. We can also identify the arguments by their name, regardless of the order. That's also why they're called keyword arguments -- instead of their position I can call them by their keyword, their name.

For most parts, this can get you very far. You can now create functions with as few or as many arguments as you need, and you can make some or all of these arguments with default values, so we can skip them or call them by their name.

The hitch with keyword arguments

Of course, not all is easy in the land of computers, so there's a hitch that we have to look out for when using keyword arguments. These arguments are only initialized when the function is defined, so they become what other languages would call static, they don't change from call to call.

Well, what do I mean by that now?! It means that when you give a keyword argument a default value that can change, like a list, a dictionary or an object, you'll be surprised. Let's have a look:

def wrong_insert(a, ls=[]):
    """Insert a into the given list."""
    ls.append(a)
    return ls

Now when you use insert, at first nothing surprising happens:

>>> wrong_insert(2, [])
[2]
>>> wrong_insert(3, [1, 2])
[1, 2, 3]
>>> wrong_insert(1)     # we can omit the keyword argument here!
[1]

However, now something completely strange appears:

>>> wrong_insert(1)     # again!
[1, 1]

Wait, what now?! Why is 1 twice in that list? Because you inserted twice into the same list. That default value is only created once when you define the function, so when you append something to the list, you append it to the default value of that parameter. So how do we do it right? We assign a default value that cannot be changed and test for it. There's one value that's perfectly suited for that task, and it's None:

def right_insert(a, ls=None):
    if ls is None:
        ls = []
    ls.append(a)
    return ls

This one works as expected:

>>> right_insert(2, [])
[2]
>>> right_insert(3, [1, 2])
[1, 2, 3]
>>> right_insert(1)
[1]
>>> right_insert(1)
[1]

Be very, very careful when you see a list, a dictionary or an object as a default value. Changing that value will happen and you probably won't expect it.

There's more or less only one use case for putting a mutable value (like a dictionary) into a default value, and that's when you use it as a cache. You should certainly call it CACHE in that case, too! Here's an example:

def fib(n, CACHE={}):
    if n in CACHE:
        return CACHE[n]
    else:
        if n < 1:
            return 1
        else:
            val = fib(n-1) + fib(n-2)
            CACHE[n] = val
            return val

That's a neatly encapsulated cache. Since we named it in ALL-CAPS, we know that it's sort-of global, so we'll be on the lookout when finding such a variable.

If you see any other list, dictionary or object as a default parameter value, be suspicious!

varargs, lots of positional arguments

But what if you don't know the number of positional arguments beforehand? Say, you want to add two or more numbers? There's a way to do that and it's called varargs, from "variable number of arguments". The syntax for that in Python is:

def adder(*args):
    result = 0
    for argument in args:
        result = result + argument
    return result

And let's try it out:

>>> adder(1)
1
>>> adder()
0
>>> adder(1, 1, 2, 3, 5, 8, 13, 21)
54

So what happens here is that any argument that is not positional gets stuffed into a list, and this list is called args, because we gave it that name with an asterisk in front of it in the function definition. Any other name works as well, but it's a pretty strong convention to use args here. And as you see, we can have any number of arguments, even none at all. Of course we can still specify that we want positional arguments in front of it:

def subber(start, *args):
    result = start
    for argument in args:
        result = result - argument
    return result

You can not omit the first argument. It's a positional argument, so you cannot omit it. But we can omit the varargs:

>>> subber(5, 2)
3
>>> subber(10, 1, 2, 3, 4)
0
>>> subber(10)
10
>>> subber()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: subber() missing 1 required positional argument: 'start'

Can you have more than one varargs?

def subber(*args, *moreargs):
  File "<stdin>", line 1
    def subber(*args, *moreargs):
                      ^
SyntaxError: invalid syntax

No, you can't. And what would be the meaning of that anyway? Which list should Python add the arguments to? No, no, that doesn't mean anything, so you it's a syntax error, as it should be.

Did you notice what simple trick Python does here for us? It takes the arguments that you pass into a function that don't go anywhere else, stuffs them into a list and gives us that list. It's really all that happens.

But interestingly enough, the trick works in reverse, too: When you have a list of values that you want to use as arguments to a function, you can tell python to do just that, and with the same syntax, too:

>>> arglist = [10, 5, 4]
>>> subber(*arglist)
1
>>> subber(10, *arglist)
-9
>>> subber(100, 1, 2, 3, 4, *arglist)
71

Don't forget that asterisk, though:

>>> subber(10, arglist)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in subber
TypeError: unsupported operand type(s) for +: 'int' and 'list'

kwargs, lots of keyword arguments and lots of other arguments

So let's say you want to define a function that has a keyword argument, but you don't know what keyword that'll be, and you also don't know how many it should be. Sounds unlikely, but it's actually very useful when you do dynamic processing or datastructures you don't know much about.

Of course, there's a syntax for it as well, and it's pretty similar to the varargs we saw before. It's called kwargs, for keyword arguments, uses two asterisks and instead of a list of arguments you get a dictionary:

def dynamic(**kwargs):
    print(kwargs)

And it works as expected:

>>> dynamic(hello='world', goodbye='lonelyness')
{'hello': 'world', 'goodbye': 'lonelyness'}

Just as we saw before, the same works in reverse: When I have a dictionary with values that I want to use to call a function, I can just tell Python to unpack them into arguments:

def do_stuff(key=0, lock=1, **kwargs):
    print("doing stuff with key {} and lock {}".format(key, lock))
    print("and more arguments: {}".format(kwargs))

>>> params = {'key': 10, 'lock': 'Locke', 'something': 'else'}
>>> do_stuff(**params)
doing stuff with key 10 and lock Locke
and more arguments: {'something': 'else'}

So now we have all parts together and we can actually stick them together, so here's the full syntax for function definitions at (least in Python 2.7):

  def fun(a, b, *args, **kwargs):
      print(a, b, args, kwargs)

Try it out for yourself!

Actually, there's one more thing: You could add in keyword arguments in the middle, between b and *args, but it wouldn't work the way you expected it to, and that would be confusing and unintuitive. So don't do that!

Functions are values!

So why is the args and kwargs dance so useful? Especially kwargs -- what good could it be when you don't know the parameters to your function?

To make this work for us, we have to know one more great thing about functions: they are values, just like everything else is. So you can rename functions, store them in variables, put them in lists and even pass them to other functions. Using fun from above, let's see:

>>> fun(0, 0)
(0, 0, [], {})
>>> morefun = fun
>>> morefun(0, 0)
(0, 0, [], {})
>>> funlist = [fun, morefun]
>>> funlist[0](0, 0)
(0, 0, [], {})
>>> def funfun(f):
        print("calling f")
        f(0, 0)
>>> funfun(fun)
calling f
(0, 0, [], {})

That last bit is really interesting: we can pass functions into functions. We can also return functions from functions -- because they're simply values that behave like any other.

And finally, we can also define functions inside of functions. In a way, when you write def f(...):..., this is just shorthand for f = <some function>, and it will behave exactly like any other variable. Think of functions as any other variable, just one that you can call to get a new value.

So now let's put all this together. See if you can figure out what this next example does:

def log_fun(f):
    def inside(*args, **kwargs):
        print("called with {} and {}".format(args, kwargs))
        result = f(*args, **kwargs)
        print("resulted in {}".format(result))
        return result
    return inside

Wow! We defined a function that takes a function, creates a function, uses a function inside the inner function, and then returns that inner function. So, what does it do? log_fun wraps the function we passed as parameter f, and returns a function that does the wrapping. The wrapping function then prints the arguments and kwarguments, calls f with them, prints the result of the call and finally returns the result of the call. All without knowing anythig about the parameters of f itself. From a parameter/return perspective, it behaves quite like f does -- only it prints out the parameters and results in between. How do we use it? Let's try:

>>> new_f = log_fun(fun)
>>> new_f(0, 0)
called with (0, 0) and {}
0 0 () {}
resulted in None

You could be even more daring and replace fun with that new logged version of fun:

>>> fun = log_fun(fun)
>>> fun(0, 0)
called with (0, 0) and {}
0 0 () {}
resulted in None

Pretty neat, eh? You simply wrapped your function in some logging, without caring what the actual parameters are. That log_fun function is pretty useful when you want to debug your function calls without changing the functions themselves.

Since wrapping, enhancing or decorating functions is such a useful concept, there's a special syntax for it in python, with an @. These next two examples are exactly the same:

def something(x):
    print("X!", x)
something = log_fun(something)

and this one:

@log_fun
def something(x):
    print("X!", x)

And now you know what a Decorator is in python: It's a function dthat takes a function f and returns a function f', and in-between does something with the original function f -- effectively enhancing the things f does. Don't be alarmed by the syntax, the two blocks above are exactly identical, semantically, so if you ever see one of the two, you can replace it with the other without fear.

There's more to say about decorators, but there's another document for that.

Overloading

If you ever programmed in a compiled language like C, C++ or Java, you might have heard the term "overloading a function". It's a cool concept, and it basically means that two functions or methods can have the same name but a different parameter list. Like this (in Java):

class OverloadingDemo
{
    void simpleFunc()
    {
        System.out.println('simpleFunc v1 called')
    }

    int simpleFunc(int param)
    {
        System.out.println('simpleFunc v2 called')
        return param * 2;
    }

    public static void main(String[] args)
    {
        simpleFunc();
        simpleFunc(2);
    }
}

If you compiled that program and ran it, the output would be:

simpleFunc v1 called
simpleFunc v2 called

The reason this works is because when you compile this program, the compiler can match the different calls to the different function definitions. So effectively, there are two functions in this Java class, one without parameters and one with parameters.

Python works differently. When you define a function, as you remember, you simply give a value to a variable. So if you do this in Python:

def fun():
    print("fun v1 called")

def fun(a):
    print("fun v2 called")

Now there is still only one version of fun, and it's the last one that you defined. The second definition of fun didn't overload the function, it replaced it.

If you want your function to work the same way as the ones in the example above, you'll have to do it yourself:

def fun(a=None):
    if a is None:
        print("fun v1 called")
    else:
        print("fun v2 called")

So, the gist is that Python does not have built-in overloading. Instead, you can assign a None value for the parameter you want to overload and check for its existence yourself.

Conclusion

Well, there's a surprising amount of stuff in Python functions. The most important things you have to remember are:

If you want a variable number of positional arguments, use args. If you want to variable number of named arguments, use kwargs.
Function definitions have a lot of syntax, but they're nothing "magic". They just simply create a function object and assign that value to a name, the function name.
You can not overload a function in the Java-sense, but you can overload them in other ways, with named arguments, args, kwargs and your own separation logic.
Do not use a mutable value as a default parameter value.

And finally, don't worry too much about it! =)

We've lost our top rating

Thu, 23 Jan 2014 14:01:35 +0100

Yesterday, Qualys released a new SSL test with updated guidelines.

Until that moment, our page at https://sixsafe.com/ scored straight A in their test. Now we only score A-. Check it out for yourself! Then check your own page! The reason for this is that the new test includes new guidelines on Forward Secrecy. For some combinations of browsers by a certain OS company, we don't provide Forward Secrecy, leading to a downgrading.

Unfortunately, there's not much I can do about this, because our web server does not support the necessary ciphers in the version we're using. So for the moment, we'll stay at A- and update our Forward Secrecy at some later point.

Mark your live deployments with userscripts

Fri, 17 Jan 2014 15:27:50 +0100

I recently launched a web service at its own domain. I am actively developing the service on my local machine, so I always have at least two instances running at the same time.

Thanks to that situation, I am scared whenever I change something that I'm doing the change to the wrong instance. Which would be bad.

So I built myself a little tool to help me discern between the two: a bright red tag that shows me which the live version is. It's really simple with a Userstyle. All I did was add a new userstyle for the domain name of my service:

@-moz-document domain("sixsafe.com") {

 body:before {

   content: "LIVE";

   font-size: 50px;
   background-color: red;
   color: #eee;

   margin: 0;
   padding: 100px 100px 5px 100px;

   position: absolute;
   top: -50px; left: -110px;

   -moz-transform: rotate(-45deg);
 }
}

You can see the result in the image: a bright red "LIVE" tag.

This solution is really great for me: It's non-intrusive, so no-one else can see it. It's easy to install without changing the service and it's pretty obvious which instance I am working with. Everything I wanted!

Gamification 2.0

Tue, 23 Apr 2013 21:45:22 +0200

A friend recently sent me a link to Jane McGonigals excellent TED talk about Gamification. Watching it, I was reminded of the Mother Of All Gamification talks, Jesse Schell's Visions of the Gamepocalypse. I really enjoyed both of these talks, but I don't like their conclusions. ⊕
Even Coursera now offers courses on Gamification. Which is fitting, I guess, since university courses have been "gamified" with points and regular feedback for a long time.[/caption]

What both Schell and McGonical want to do is make games that "trick" the player into doing something with himself or his surroundings at the same time. I don't think that's going to work. For one, because it's dishonest, not telling the user what you're actually trying to achieve with the game. But more importantly, because they are diluting the game design and experience with their ulterior motives. Did you play educational games when you were young? Probably Math Blaster! or Mario Teaches Typing? We had some of these when I was small and I never liked them. The reason is that you're playing a nice little game and suddenly something interrupts your flow with a math question. That's not a fun game mechanic. That's not any game mechanic, actually, that's just an annoying block between you and the game.

There is a second mainstream meaning of the word gamification, which is "turning some task into a game". Wikipedia has a nice list of things that belong into this category, i.e. things that people have tried to do. I think that's not the right way either, because I don't think that achievements (as funny as they may sometimes be) are the way to use the knowledge and power we have learnt from games to make the world a better or easier place. I also think that achievements, specifically, are a pretty poor game mechanic that's been used in recent years because they're so easy to make (however, hard to get right), in contrast to meaningful in-game achievement without a badge, which is pretty hard to manufacture.

However, I think that gamification is something quite different: Gamification is using the knowledge of game mechanics and techniques and how they work, and apply them to different areas than computer games. Note that I am speaking only of knowledge of game mechanics, not game mechanics themselves. What I think games get right is finding and mining human traits like motivation and persistence. Games are exceedingly good at that, and good games even more so.

Leading

There's a nice explanation on Youtube how Mega Man X gets it right. And did you know that the iconic first level of Super Mario Bros is designed exactly to show you what you can do in the game? Jumping, hitting blocks, getting coins from � blocks, enemies that are dangerous and that you can defeat them by jumping on top of them. All of that from one single screen, and in less than ten seconds. Without a single text display.

That is great gamification. They "gamified" their intro/tutorial and showed the player all these things effortlessly. Designing that level must have taken very long, but for the user it's so incredibly smooth. So that's one component of gamification for me: show instead of tell. Let the customer find out what's possible on their own, without bugging them, by making it easy, almost obvious what and where things are to find. Let's call this property "Leading", because a properly gamified systems leads the user towards actions that are beneficial.

Motivation

The second component of gamification that is most widely discussed is Motivation. ⊕ Points: 000000
Achievement locked
Can you scroll to a million points? Will you?

Let's stay with Mario for a second: Mario has a coin counter and a points counter, and they are constantly increasing, all the time. Every time you do something, you get points or coins or both. So, in the simple gamification models of today, the designer would say "we need to motivate the player feedback to do something, so let's add points to it!" That has been done, to great effect, and I guess it can work—for the right kind of person. But I'm not that kind of person, and to me it feels very childish to attach meaningless points to meaningful tasks, which is exactly what Jesse Schell points at in his wonderful talk. There are systems, games, out there, that try to do this sucessfully (by Miss McGonical again), so they demonstrate that it works for some situations and for some people. However, the point counter in Mario isn't really the reason why one continues to play, is it? The point counter is merely an indicator that you're going in the right direction and how far you've gotten. As such, it's great, but it's not the motivator. The motivator is something different: curiosity, urgency, "just one more level", epicness, or something like that. This is what we have to take from games into the real world, not the stupid point counter. Of course, taking something so fluid and hard-to-grasp like motivation from a game and modelling a real-world task with that in mind is very hard. So it's natural to try out simple schemes first, like the point counter. But I think that a properly gamified system should motivate the user through its properties and its whole, not through meaningless points or achievements. So, let's take Motivation as our second component.

There are a myriad ways to model and channel this motivation, like levels, stages, medals, progress, goodies or, yes, achievements. In fact, there are so many that finding the right combination for a specific situation can be pretty hard.

Another important concept that helps form and channel user Motivation are reward schedules. And these are actually being taken into consideration when doing gamification. Some people proclaim that gamification should make tasks in the real world harder, but I don't think that's the case. Rather, it should move the challenge and the reward closer together, but keep the overall challenge/reward levels about the same. Which, thanks to human psychology, actually makes the task easier to do for a person. Brushing your teeth is a great example here (and the reason why Jesse Schell used it), because there's a challenge and a reward and they are very far apart from each other. The challenge is to brush your teeth twice or thrice a day. The reward is that your teeth will not develop problems and pain for you at some point in the future. So in this case, the world we live in is badly gamified, because the effort has to be put in right now and frequently from now on, while the reward is uncertain and a long way away. Humans are just not good at that kind of schedule, so here's something to be optimized. However, changing the reward schedule for brushing your teeth doesn't make it any harder or easier to do, it just makes it more natural for humans to do. In a way, a good reward schedule makes it easier to mine the motivation a person has for any goal and to apply that motivation in the right direction towards achieving that goal.

There are very simple ways to do that and very complicated ways, but there are ways. I think we need to be getting better at finding them. Still, Motivation, one of the central points of Gamification.

Feedback

The third point I'd like to add to the essentials is Feedback. Humans love feedback and they need it to function properly. Giving people feedback on any process allows them to get better at it, so that's what they will do naturally. Let's look at our two examples from above. When you brush your teeth, there is some feedback that you did well: a fresh breath, smooth teeth, and a whiter smile. If you skip once or twice, that feedback will deteriorate, so once you stopped brushing your teeth the feedback won't work as well as before. However, there's also very little clear indication about how good you did. There's no way to check that you brushed everywhere, that you brushed enough and that you brushed for long enough. So the feedback loop isn't very good, and that's probably the reason why you don't know anyone who's a master a brushing their teeth. Now let's analyze Mario. In a game of Mario, every action you take has a clear and consistent feedback value. Jumping on an enemy: happy sound, enemy dies and you get points (and sometimes even a shell). Bumping a � block: points, coins and another happy sound. Walk into an enemy: sad sound, your character falls of the screen and a live is deducted. Note that all of these actions are visible and clearly good or bad. If you do something that's very good, like finishing a level, a small fanfare is played and you get to see a nice little cut scene. Playing Mario is one giant session of trial and feedback. Some people say that positive feedback is all that makes a game good, and I tend to agree: give a person feedback on what they're doing and they'll get better and better at it.

Most systems we use today are not very good at this. This is sad, because there are so many simple things one could add to most systems to make them give proper feedback, like status display, diagrams, scoreboards or suggestions for improvement. People can master any system as long as they are shown whether what they're doing is good or bad.

Narrative

⊕
Our hero begins his quest with a vow to finish this article. There is a fourth point that is central to any gamification effort you want to undertake, and that is narrative. People crave narratives of any kind. So much in fact, that they will create their own stories for something as simple as a 32 x 16 monochrome display, showing a state machine with less than 16 states, making them bond with that state machine and get up in the middle of the night to 'feed' it. Note, however, that Tamagotchi is an excellent distillation of Feedback and thus masterfully gamified: people want to carry the toy around and care for it every day, no matter its simplicity, creating Motivation and Narrative out of the players themselves.
Some of the most well-known and loved toys and games of the world are simple playgrounds for creating narratives: Lego, Barbie, Playmobil, The Sims and, of course, Minecraft. All these games offer is a rich environment for creating your own stories and adventures.
At the same time, even the most simple-minded games offer some kind of background story, whether that may be furious birds or supertechnology soldiers.

The truth is that Narratives make it easier for people to understand a situation and muster up Motivation for it, even if the Narrative is quite thin. Even if you know that the princess is in another castle, here's a good and simple reason to slog through all these complicated worlds: find the princess and rescue her. Be a hero!

Sometimes, team selections are used to create a simple but powerful Narrative: us against them. That seems to work for all recent 3d shooters, even if the teams are absolutely identical.

In a way, Narrative is also something that McGonical and Schell point at and want to use as a tool to motivate the player. It's just that the methods they choose (point counters and badges) are quite crude and not very well-fitting, in my opinion. However, it's also true that Narrative is probably the hardest to add to any Gamification project that is not actually a game. How would you fit a Narrative (that is not totally moronic) around brushing your teeth or doing your taxes? I don't know! But figure it out and you'll have the power to help people do something important. And you'll find easy ways to collect your reward for that - even if it's just the knowledge that you're a hero!

Conclusion

So, to recap: Leading. Motivation. Feedback. Narrative. Add these four to any system you are designing, and you will have a properly gamified system that will work very well.

It's hard to tell how to actually use these things in any real product, but I think it's still worthwhile to start thinking about gamification on a broader scale.

OAuth and Django — a cautionary tale

Fri, 01 Feb 2013 09:07:00 +0100

Have you ever implemented OAuth authorization in any project of yours? Well, I have these last two weeks in a Django project I am doing, and it was not a nice experience. The following is going to be a bit of a rant against the state of authentication libraries, but take it as a cautionary tale and remember one thing:

Hours of doing can save you minutes of thinking.

So, let me tell you a little something about my experiences.

The first thing you'll encounter on your quest is the choice between OAuth 1.0 and OAuth 2.0. There are voices for both sides, and they're both very convincing, so what you're going to do (if you're like me) is first implement OAuth1, then switch to OAuth2, then scrap that and go back to OAuth1. Because the first thing you'll try is to find and use a good OAuth1 library for Django, of which there are none. Oh, there are OAuth libraries all around, but I couldn't get any of them to work. So you'll say to yourself, screw this, I'm going to do OAuth2, it's better anyway. Then you'll spend some time reading up on bearer tokens and refresh tokens and grant types and you'll come away more confused than enlightened. Then you'll play around a bit with the existing implementations like Google's wonderful OAuth playground and you'll say to yourself: I can do this! And you'll start and do that, reading parts of the humongous spec on the side and notice more and more that there's loads of stuff in there that you don't even want to think about. Then you'll not get it done right, you won't find any client libraries for using it either, and then finally you'll read Eran Hammer's rant over OAuth2 and decide that it's really not in your best interest to implement OAuth2, and go back to searching for OAuth1 libraries you could use.

So now you're stuck at that part again, but this time, you're determined. You're determined to find a stable and good implementation of OAuth1, that also offers 1.0a, because that's important for security. You'll skip oauthlib because you don't trust it (a mistake, as you'll find out later) and also because the documentation is a bit sparse. Well-written but sparse. So instead you'll go directly for Leah Culvers six-years-old OAuth implementation. It's a bit crummy and you can see its age, but it's quite simple and adaptable, so you copy it verbatim into your project and start hacking away…

A mere two weeks later, you'll have touched every part of the library and added your own crumminess to it. You'll have an OAuth implementation that's half-working, is neither OAuth 1.0 nor OAuth 1.0a but somewhere in between, and you'll find new errors every other day, because you never took the time to read any of the OAuth tutorials there are — because you really couldn't find good and simple ones, detailing the differences between OAuth versions. But, you'll have a working authentication scheme that uses signed requests and never exchanges secrets in the clear, which is a good thing all on its own.

So let's examine the issues you will have had by the time you've come this far.

You did some reading at the beginning about OAuth and its versions, but couldn't find good enough material to make a decision. Go back and do that now, you'll regret it many times if you don't. Then pick one version and (in the case of OAuth2) one workflow and stick with that. You can extend later, if you need to.
You didn't find any good, trustworthy-looking libraries, for neither version of OAuth. That's ok, just pick one and stick with it. Switching between libraries will not do you any good but burn your time away.
Once you've collected some experience and read enough of the materials and the spec, write down your findings and make it easier for others and for future-you to find them and learn from it.

As you can see, I'm in the third stage described. I've tried to address some of the issues I've found with current OAuth implementations and collect my knowledge about that stuff in code. It's far from finished and far from done, but it will be useful once you start an OAuth1 implementation for yourself. You can find that code on GitHub along with some documentation I wrote, and I sincerely hope you can learn from my mistakes.

Simplicity

Thu, 08 Nov 2012 15:05:00 +0100

A few days ago, a colleague pointed out to me that "simplicity" is not a feature. His reasoning behind this was that simplicity means different things for different persons, so simplicity can mean anything inside of a range of things.

I don't agree with his statement at all, and I actually think that simplicity is the only feature any program can have. In the widest sense, all that computers do is provide a given task in a simpler way, usually making that task faster and more accessible to different groups of users. Some of the times, making a task simpler makes the task possible in the first place.

Even at the beginning, one of the first instances of a computer was the famous difference engine, invented by Charles Babbage. Its sole purpose? Computing tables of logarithms in a quick and reliable way. In other words: making finding a given logarithm simpler for the user. Note that the logarithm tables themselves are a way of making that task simpler in the first place.
After the difference engine came programmable looms by J. M. Jaquard, which made the production of complicated, patterned fabrics simpler.
Later, "real" programmable computers were invented by Konrad Zuse in Germany and von Neuman, Turing and others in Great Britain. Their use cases? Solving linear equations, doing the necessary computations to reverse code ciphers and computing ballistic tables. All of which make the underlying task simpler for the user. They also made these tasks a lot quicker and a lot more reliable, but these are merely side effects of making the tasks simpler for the user.

Mainframes: Making it simpler to do complex calculations of any kind for many users.

Home computers: Making it simpler to do complex calculations of any kind on your own.

The internet: Making it simpler to retrieve information that is stored in different places.

Email: Making it simpler to contact people that are not immediately available (i.e. making it simpler to send letters).

Even "modern" services and gadgets like Google (making it simpler to find some websites that store information you might be interested in), Facebook (making it simpler to have a homepage, share photos and see what others are doing), Twitter (making it simpler to tell everyone who's interested what you're doing) or Apple computers (making it simpler to use all the awesome stuff that nerds built for themselves) are built on that one guiding principle of simplicity.

Note that oftentimes, with simplicity come other benefits like scalability or even availability, which help adoption and development of even newer, even simpler products.

But the main thing that software, that technology does, is making stuff simple. I can't think of a single example of a product or service that is not making something simpler.

Shameless plug: I'm doing Bit Chest at the moment, which is built around being a simple and safe file archiving solution.

Over-specialization is the curse of our civilization

Tue, 06 Nov 2012 07:25:00 +0100

I'm sick of the distinction between blue-collar and white-collar workers.

The blue-collar people think they're hot shit because they solve the hard problems. They know how to build. They look down on white-collar workers who are only solving trivial problems. Blue-collar workers build the real things. They don't have time to learn about spreadsheets or a wretched scheme like Wall street. Without them, a nation can never be successful.

The white-collar workers think they're hot shit because they're the closest to the trade. They know to perfect a business process. They look down on blue-collar workers who have no idea how the produce is actually used. White-collar workers build the real things. They don't have time to learn about crop rotation or wretched schemes like shift work. Without them, a nation can never be successful.

You're both wrong. You must do both or you'll be irrelevant (somewhere).

Over-specialization is the curse of our civilization. Stratification breeds condescension. The only remedy is for all workers to do everything.

This post is an answer to the similarly incendiary Over-specialization is the curse of our profession

Great presentations: Steli Efti on Happiness as an Entrepreneur

Thu, 09 Aug 2012 09:02:00 +0200

I just watched Steli Efti's TEDx talk on happiness as an entrepreneur.

It's a great presentation: moving, emotional and with a great message.

However, if you look only at the technical aspects of his talk, it should have been quite mediocre. He didn't use any slides, he walked around all the time, he had to hold his own microphone. All stuff that we're told is bad for a presentation. But still, it was really good to look at and go with it.

Why? Because Mr Efti has a message that he cares about. And that is the only thing that can make a good presentation.

The last of the first

Fri, 13 Jul 2012 21:11:14 +0200

I just threw out the rest of my first start-up's stuff. It was a big box of flyers we had printed and wanted to distribute among students. Most of the flyers had handwritten codes on them that would allow people to try out our product, http://partycode.de/ for free. We hand-wrote them because it was cheaper.

Back then, I wasn't afraid of ordering 5000 printed flyers. I also wasn't afraid of throwing out the first batch because I had a spelling error on them, and ordering a new batch. I wasn't afraid of an evening of writing codes with friends.

But throwing them out...

I had kept the stuff for a year, before I finally liberated myself. It's not that I didn't know that it was over, after all, I had shut down all work on that enterprise, as well as the next one, but still, throwing away all that stuff that I worked so hard for felt strange.

Validation

Wed, 16 May 2012 09:53:38 +0200

You know the situation: You've written a great new product, you've just released it (much to your own relief, everything seems to be working) and now you're contemplating the next steps.

A little while later, you've done your homework, you have a Facebook page, you have a Twitter account, you have put your release out there on Reddit and Hacker News and StumbleUpon and even Delicious. And somehow, nothing happened. Maybe you've sold a copy or two, but annoyingly, the masses don't seem to be noticing your nice little App at all.

At first, your spirits don't mind. These people, they'll come! They'll find my pages, they'll notice my marketing and see my ads. Maybe after the next version? Maybe if you tweak your analytics just a little bit more...

Some days pass, and nothing happens. Your spirits really do start to mind, and indeed they fall at the same rate that your helplessness rises. You sit around sulking and think about what you did wrong.

And all of a sudden, this happens:

At that moment, you know what's missing: Validation! You needed someone to tell you that some of your stuff is interesting. That at least someone is looking at it.

Now, you can continue!

Simple APIs

Thu, 26 Apr 2012 20:54:23 +0200

I like simple APIs.

Simple in the sense of small, with few options and few available calls, as opposed to monstrous can-do-everything-APIs. Alright, so maybe I should say small from now on.

Before I tell you why I like small APIs more than bigger ones, let me first tell you why I don't like large ones. There are actually severe drawbacks associated with larger APIs and a bigger selection of available methods and options, but not many people seem to notice, or care.

The first thing that large APIs do with me is make me feel lost. That is obviously true for really large APIs (like the MFC, with which I had first contact when I was fifteen and which probably delayed my programming career by a few years), because there's just so much to see and so much to do that you suffer from the well-known choice paradox as well as sensory overload -- you simply don't know where to look at first, so instead of finding an entry point, I just gave up. But this also applies to smaller libraries where many options are exposed that look like very similar at first glance. Case in point here is the wonderful ELCImagePicker, that I'm using on Imprys. I don't want to bash on this library, it's quite excellent indeed, and I would have had a hard time implementing it myself.

However, and this is the second things that bugs me about large APIs, every time I use the ELCImagePicker, I am left unsure whether I did everything right. Instead of providing a single entry point with a single method and a single callback protocol, I have to compose two classes myself. Two classes. This is about as simple as it gets, but still, it's more complicated than I would like.

Call me lazy (and laziness is probably one part of it), but the more important part is decisions. Programs and program libraries are collections of decisions. The more decisions are encapsulated in a library, the more I can feel safe that these decisions are right in that context, because someone (and probably someone smarter than me) thought about that decision, weighed the pro's and con's of each possible option and then just made the decision. So I don't have to.

This is why I like small APIs: They make me feel safe in my choices.

So what can you do about large APIs? That question is actually quite a philosophical topic, because in a way, each component in a computer program, down to the language you are using, is an API. I think that the only way to handle each and every of these APIs is to make your own, for the situation that you're in. Wrapping lower-level APIs, more or less, because that's what writing programs is, anyway.

So, create a DSL, even if that DSL is "just" a class with two methods. In fact, I think that a class with two methods is probably a really, really good API, and if it fits your program, you should use that. Your program should look like this:

#import "mylib.h"
int main()
{
  mylib.choice choice = mylib.mainmenu();
  mylib.mainprogram(choice);
}

That, right there, is an awesome program. It's powerful, it does what it's supposed to, I can understand it. And it uses all of its underlying APIs.

And that's why I like simple APIs.

Starting up again

Wed, 25 Apr 2012 18:37:26 +0200

After my vacation and another week of hanging around, I started up again today. Feels good to be back in the programmer's seat.

Imprys can now do graphics, although they aren't saved in the right format yet, and they don't get exported to HTML, neither. Before that can happen, I will have to restructure my output writer to use a templating system, most probably the wonderfully simple Mustache.

One more annoyance: I've tried to use the Apple-supplied UIImagePicker. There are some strange design decisions in there, for example, in the iPad, you can only show the UIImagePicker in its own UIPopoverController.

Also, it doesn't allow picking more than one image at a time, which feels a bit silly when you try to add ten or twenty images to your presentation. So I've looked around the web and, behold, there is an alternative implementation on Github, ELCImagePickerController, which works beautifully. Until the moment when you find out that this image picker needs to have the Location API enabled and needs access to it. Mind you, none of the images on my iPad is location-encoded, because I have locations disabled. Still, to read the album headings, location API needs to be enabled. Which is a bit of a shame, because for those users that don't have that enabled, I'll have to fall back on the regular UIImagePicker.

Charlemagne: Keep the fire burning | The Economist

Thu, 06 Oct 2011 08:26:00 +0200

Charlemagne

Keep the fire burning

Why Germany seems not to want a quick fix for the euro crisis

TO HEAR Germany say no to fiscal stimulus, no to boosting the euro zone’s rescue fund, no to joint Eurobonds, one begins to wonder: does it really want to resolve the euro zone’s crisis? Of course we do, most Germans would reply. Is not the chancellor, Angela Merkel, the first to declare that “if the euro fails, Europe will fail”? Is Germany not the main provider of rescue loans to Greece, Ireland and Portugal?

All this is true. Yet Mrs Merkel seems to lack a sense of urgency. Despite the world’s calls for action, she does not believe in bold strokes—be it letting Greece default, or issuing Eurobonds to mutualise governments’ debt. Only a slow, step-by-step approach will work. In other words, the pain, austerity and market turmoil will go on for the foreseeable future.

Economist.com/blogs/charlemagne

via economist.com

Even though it pains me to bring attention to a rather silly opinion piece on the Economist, I just have to point out a few things about the piece linked above.

First of all, having a sense of urgency is not the same as panicking, which, to a German, is how many of the bold-stroked measures implemented by the USA or Great Britain feel like. "How much money can we find to just give away?" is not a position I would like my government to take.

Second, I object to the off-handed description of Germanys institutions. The "complex federal system" isn't any more complex than any other federal system, like, say, that of the USA. A sceptical public is a good thing for a democracy. Coalition politics are about finding compromises and will only appear messy to someone coming from an either/or-system, which feels very primitive to many Europeans. And finally, "jealous institutions such as the constitutional court"? Really? Jealous? A working, well-respected institution that upholds the central law code of a country is "jealous"? This, as well, coming from a resident of a country where the constitutional court feels like just another venue for political puppery, doesn't actually feel like an insult and instead more like a praise.

Third, there is growth. I don't want to comment more on that topic than saying that "X is weakening growth, thus X is bad" is not a valid argument until you have proven that growth is important. I know that many people assume this automatically in the United States, but it's not a given in other places. And, at least from time to time, you should think about it.

Then there are the weak arguments. Charlemagne quotes that one problem with seeing slow steps as a problem "is that market panic can become self-fulfilling, in many different ways". This is supposed to be an argument against a transaction tax. A tax that would temper the possibility for panic and create a cushion for when it occurs is held as an argument against that tax.

And, finally, there is the obsession with the german language. Yes, "Schulden" does share the same stem as "Schuld", just like, for example, "candidate" and "candid" share the same stem, even though most candidates aren't. Or "champion" and "Kampf" (which means struggle or fight). Language is a strange beast and goes wounded ways. Bringing up the similarity between Schulden and Schuld (which, by the way, also means "debt" in this context) feels like filling up for missing real arguments. Oh, and also, bringing up the protestant past for a country where less than half of its inhabitants believe in any spiritual force isn't really a strong point, either.

Actually, the more I read this piece, the more I get the feeling of a very strong cognitive dissonance. On the one hand, Germany is bad for letting "the fires burn", on the other hand, Germany seems to be very stable and financially secure at the moment. It's sad that even such a careful observer as Charlemagne seems to be likes to forgo arguments for rhetoric once in a while.

A very nice experience

Thu, 29 Sep 2011 21:16:55 +0200

Today, I joined Bitbucket. A friend invited me. And, I must say, I am very pleasantly surprised. I have used Github before, but I just can't get warm with git, and of course that colours the experience for Github.

However, the experience for BitBucket is also very friendly, and very simple. They even send you a welcome-email with pointers to the next steps, including a tutorial for newbies and a tutorial for Subversion switchers.

And one factor, naturally, is always price. BitBucket has the best price you can get, because it's free. Other than GitHub, you can have as many repositories as you want, and as long as you don't have more than five collaborators, everything's free.

If you're interesting in trying out a DVCS, or find a nice host for Mercurial, I urge you to try BitBucket.

Foreign code

Thu, 22 Sep 2011 10:25:51 +0200

On last weeks' Sea Of Memes the topic of using libraries, working together with other people and copyright issues came up.

Then, a few days later, Shamus Young discussed the article on his blog Twenty Sided.

I disagree, with both of them, on almost all the issues raised.

The first issue is "working with strangers". Here's the quote:

I'm also trying to learn things about graphics programming and game programming. You don't really learn something unless you try to do it yourself. Using a big multi-platform library or an existing game engine wouldn't teach me much.

Then there's the problem of debugging. Things go wrong and you need to isolate the problem down to a particular piece of code. It helps enormously if you've written it all yourself. If other people are working in the same area, it's practically impossible to know whether the bug was caused by the thing you just changed, or by someone else's change. It also helps that I know my own style and the kinds of mistakes I tend to make.

I commend mgoodfel on his journey to learn an interesting skill and work his way towards a fully self-written game engine. I'd like to do that myself some day, so I understand that he wants to write most of the code himself. That shouldn't stop him from collaborating with other people towards this goal, however. In fact, most of the things I learned didn't come from myself sitting in the basement doing it alone, but from talking to other learners about it and talking them through my thought process, oftentimes finding crucial flaws in my thinking. Of course, listening to someone else's thought process and finding their flaws, all the while seeing a wholly different position. In a way, Sea Of Memes is mgoodfels way of achieving this, whereby he writes down his thought process, then waits for comments from his readers. He even states this specific goal in Part 29:

The writeups revolve around the code. A part gets done when I get a piece of code running, not on any regular schedule. And then I wait for feedback, either downloads of the demo, or comments or both.

But still, to me, it doesn't feel like collaboration, it's more like an improv performance, where the audience is allowed from time to time to yell a word or two about the next piece mgoodfel should perform and how it should look like.

What's even worse, there is a simple system that would address the issues cited above, as well as mine, and that is GitHub. The workflow for an open-source GitHub project is like this: You create a repository, then push your initial state there. Everyone can see this code and do a "fork", i.e. copy the repository into their own account, then check it out and work on it. Meanwhile, you've worked on it yourself, pushing code to the public front end only at times you choose (kind of like putting them into a Zip file on your blog, only that it's easier to see what changed). If someone else wants to take your code and work towards a different direction, that's fine because you won't even notice. If someone wants to contribute to your code, they push a change that implements what they want to show/give you and open a "Pull Request". This means you get a message that someone wants to contribute, a patch in regular diff format and a button that says "Do it!". You'd then read through their patch, thinking about the consequences the integration has for your code, thereby making it your own, and if you're satisfied with the integration, you pull it into your repository, merging the change. If you're not satisfied, you simply ignore the request. Happens all the time.

The point I'm trying to make is this: Pull Requests open your code up for collaboration if you choose to integrate. If not, you still saw someone else's thinking, which is always a good thing.

Oh, and you know what: The internet doesn't care when you're awake. It doesn't matter when you pull, when you push and when you work on your code, because everything is asynchronous. You and all of your potential collaborators, work at their own times, paces, and work styles.

The second issue that comes up is libraries. Here's what Shamus Young writes about using libraries:

Using code from strangers is always a gamble. Sometimes, on rare circumstances (perhaps only a few occasions in your career) the clouds will open and you’ll find yourself in a beam of sunlight, looking at a link to the perfect library while angelic music plays. It does happen. But more often than not, you run into the usual chain of dependencies, missing files, lack of documentation, version incompatibilities, inscrutable bugs, missing features, and poor programming interface.

To me, that sounds as if Mr Young was burned by some libraries or code snippets he tried to use for his own project, Project Frontier. It sounds as if he sees libraries as a time-saving tool. When he needs some functionality in his program, he thinks about the quickest way to achieve that functionality, which might be (a) writing it yourself or (b) taking code from someone else to do what you need. If you see it like that, then yes, every library is a gamble, where you wager time and hope to win back that time by not having to write the code yourself, i.e. you invest some time into integrating and probably bugfixing and working with the library, and earn back the time by saving implementation time. If the library is good, you have made a net time win, where you have the functionality quicker with option (b) than option (a). If you find out that the library isn't for you and "give up", your investment is lost and you'll have to start anew.

That's a bleak way to look at libraries, I think. To me, libraries are about focus, and not losing it. To me, the choice isn't "how much time will this cost?", instead it's "how well does this fit into my vision?". So, I ask myself: "Do I really want to write a JPEG loader?" and "Do I really want to mess with certificates myself?" (a topic I will discuss in the post after this one), or "Do I want to write a Minecraft chunk loader?". Some of the times, the answer is no, some of the times the answer is yes. When the answer is no, I'd rather spend an hour or two banging some library in shape and massaging my own code instead of looking at byte-descriptions and file signatures, until the question becomes "Do I want to maintain this ****** library or find some other way?", when I probably give up.

The third issue mgoodfel brings up is copyright and patents. This is a hairy one, and I can understand that he's afraid of being sued. On the other hand, what could anyone win from suing him? He doesn't appear to have a lot of money and if you wanted to stop him from distributing or using some piece of code, it'd probably be more effective to ask him. Most of the disputes in that area are settled out of court, anyway, and I don't think one should be scared by the way big companies with huge cash reserves behave towards each other. Also, note that Samsung and Apple are still cooperating on other areas of their businesses.

Perhaps I am too young, or haven't been burned by issues like these, or I am in a jurisdiction where it's easier to handle, but I don't think a programmer should care too much about patents, lawsuits and unintentional breach of copyright. In fact, my feeling is quite eloquently summarized by pud, when he writes "Fucking Sue Me".

So, there you have it. My personal, completely unfounded opinion on why you should do everything differently than you do. Note, again, that these are my opinions, my suggestions for thinking about it, not requests that anyone change anything about the way they do what they want.

What’s wrong with this code, really? | CV Mountain.com

Fri, 16 Sep 2011 17:25:00 +0200

via cvmountain.com

A very nice example of bad, but correct code and the rationale for replacing it with a cleaner, better version.

A powerful demonstration of Ghostery

Thu, 15 Sep 2011 08:40:03 +0200

I have used the excellent Ghostery for a few weeks now. What it does is it blocks a long list of web tracking and ad network services to make your browsing more private. Every time Ghostery blocks a service, it shows a little bubble with a message what was blocked. This way, you can easily see, who's tracking your browsing how. Also, it's simply interesting.

For example, this is yahoo.com:

google.com doesn't show any warnings, so I guess they track you on their servers, which you cannot block, of course.
For comparison, here's techdirt.com, anandtech.com and engadget.com

It's amazing that techdirt places a whole ten different services on their page that can track you. However, the funniest thing happened today, when a website I visited used Clicky (getclicky.com). I hadn't heard of it before, so I went and took a look. Unfortunately, they use similar addresses to their tracking scripts on their web page, which Ghostery blocked. The feature comparison chart is especially nice:

Well. That's a hard choice, there... =)

Focus puller - Wikipedia, the free encyclopedia

Wed, 14 Sep 2011 23:30:00 +0200

From Wikipedia, the free encyclopedia

A focus puller, or 1st assistant camera operator, is a member of a film crew’s camera department whose primary responsibility is to maintain image sharpness on whatever subject or action is being filmed.

“Pulling focus” refers to the act of changing the lens’ focus distance setting in correspondence to a moving subject’s physical distance from the focal plane. For example, if an actor moves from 8 m away from the focal plane to 3 m away from the focal plane within a shot, the focus puller will change the distance setting on the lens during the take in precise correspondence to the changing position of the actor. Additionally, the focus puller may shift focus from one subject to another within the frame, as dictated by the specific requirements of the shot (cinematic techniques).

A good focus puller will have an intimate knowledge of cinematographic and optical theory. Depending on the parameters of a given shot, there is often very little room for error. As such, the role of a focus puller is extremely important within the realm of a film production; a “soft” image will, in most circumstances, be considered unusable, since there is no way to fix such an error in post-production. One must also consider that an actor may not be able to duplicate his or her best performance in a subsequent take, so the focus puller is expected to perform flawlessly on every take. Because of these factors, most production personnel consider the focus puller to have the most difficult job on set.^{[citation needed]}

Retrieved from "http://en.wikipedia.org/wiki/Focus_puller"

via en.wikipedia.org

Today I learned that there are many, many people, measurements, time and work involved in making sure that films stay in focus. It's a lot more complicated than I thought and explains long filming schedules and budgets. Amazing!

Software’s Receding Hairline - raganwald's posterous

Sun, 11 Sep 2011 10:07:00 +0200

via raganwald.posterous.com

Interesting comparison between a comb-over and software development in the sense that both are the result of years of waiting for the right moment to change something.

Today I learned something

Thu, 01 Sep 2011 13:09:00 +0200

So, yesterday was a good day, because I finally got a push to do something about my lingering ambitions. And that's what I did today. In the process, I learned that the MacOS X is not as stable as I thought.

As I was playing around with a simple application for monitoring the system clipboard (called Pasteboard in MacOS X), I managed to corrupt the clipboard. After my application quit, any other application that would try to access the clipboard would go into an infinite loop.

I have put some proof-of-concept code on GitHub, under https://github.com/shezi/ClipCrash

The Million-Dollar-Question and its consequences

Wed, 31 Aug 2011 23:03:00 +0200

[...]
80% of the people I draw up plans for – plans that would clearly work – don’t act on them.

It’s like everyone fantasizes about… whatever… but once their fantasies start to become reality, they piss their pants and self-sabotage.

[...]

via sebastianmarshall.com

I just read this very poetic and well-written post by Sebastian Marshall about why we don't act on our urges to do what we need to do to be successful. Everyone who has ever thought about being an entrepreneur or "do something cool along the way" should read it. And while you're at it, you should read some more by Mr Marshall, because he writes an awful lot of good things.

The piece resonated with me because I am the same way. You probably know the feeling, thinking about this passion that you have, that crazy thing you always wanted to try. You get that suppressed smile on your face and you can almost not stop talking about "the thing", even though everyone thinks you're crazy.

Perhaps you've even made a plan, how you would make enough time while still working. How you'd make sure that you can pay the rent every month, because you're not crazy enough to do that kind of thing full-time. And it might even be a good plan, with numbers and dates and milestones and possible problems and all the other adult things a plan needs to have to persuade anyone that the thing you're planning is not entirely crazy.

You might even have started to work on your plan, perhaps a prototype, or technology research or market research. You might have the numbers to prove that you can do it and that you'd earn money with it.

You might even have talked to others about the plan, perhaps found collaborators, co-conspirators. You might've had a few appointments, talked with them how great it would be and how easy it would be to implement and just do it. You could have had a few beers on the occasion, celebrated your courage and strengthened it with your future business colleagues.

But then, something else happened instead. The first milestone came and went.
The prototype was never made into a real product, or perhaps it wasn't even finished. Or, you know, something in the family came up, or you really needed some safe money, right now, because your car broke down or something like that. Or maybe the kitchen needs fixing, or one of your friends asked for help with moving. Or, hey, look at that, there's already a new set of pictures on reddit that you could look at all day.

I know how that goes. Because I did.

Let me explain!

Almost six months ago now, I quit one of my jobs. At the time, I had two half-time employments, and when one of them didn't work out as planned, I took the plunge and quit. That wasn't an easy thing to do because I lost quite a lot in the process, but I did all the things I suggested above. I had dreams, and ideas, and plans even. I had talked about it with my girlfriend and my parents, and they were all for trying it. I even talked to my girlfriends father and mother about it, and even they were in favor of trying my hand at being self-employed. And so I did it. Since I only had to work 20 hours a week, which would cover all my living expenses nicely, I even had a lot of time on my hands to be a successful entrepreneur.

And, you know, I even went so far as to create a product, Partycode, which is still live and reachable, and which has, to date, not had a single customer. Because when it came to marketing, telling people about the product, going all-in, so-to-speak, I stopped. And so did all two of my fellows, even though we had had great plans for the product.

But, I wasn't deterred. I still had the time and I still wanted to do it. So I drew up more plans. Forged new alliances, found something really neat that I still believe would have worked, if I just could bring myself to do it. And so this wonderful idea, this great product, has never seen the light of day, except in the fragments of our design documents.

What do I do instead? Well, there was the Steam Summer Sale, which was nice. Oh, and then there were all these parties, and the big family event we held. I might have watched one or two TED talks, and perhaps some OpenYale courses. And, of course, Hacker News and Reddit are always there for me when I don't want to do anything else. Nowadays, it's so easy to distract yourself from this lingering thought that you should actually be working...

But, at least I think I can add something to Mr Marshall's million dollar question, because, for me, I think I know the reason.

The reason is potential. Right now, I'm a potential entrepreneur, a potential genius and millionaire and the ideas I have are potentially wonderful, successful and great, and potentially, everyone loves me for producing such a great thing. Actually starting to work on the ideas rubs that golden glow off and makes them less potential, less potent even, although in the process, they would get real. It would take away the fairy dust and add adult things, like milestones and feature lists and press releases, things that don't fit into the perfect vision of the ideal elegant product. And it would also carry the risk of failure. All of a sudden, you're not a millionaire, instead you're just a programmer who has tried to be more than he thinks he is. You're just a nerd than cannot understand why other people are not as nerdy, why no-one would even want to try out revolution you produced. You're just an idiot with no idea about marketing apart from Google AdWords. You're just a lazy guy, sitting at home all day, thinking about how great he might be one day, without ever trying. You're just, let's face it, a loser who should go back to working for the man, because the man knows better anyway.

It's not that easy to face your dreams' destruction. Perhaps even more so when some of your ideas have already not worked.

But writing all this down, I realize that it's worth it. Sitting on your dreams until you're old and grey, letting time get away from you, that's not the way to go. I don't want to look back at my life with rueful eyes, thinking that I _should_ have tried it. I want to look back, thinking: I did it.

So, I'm going to start following my own advice. In a first, painful step, I deleted all the games on this computer and blocked some websites that might distract me from the things I really want to do.

And I'm going to start posting here, every day. Come back tomorrow and you'll see...

::Maximise:: - weblog of Max Tobiasen

Wed, 31 Aug 2011 21:33:00 +0200

Moving a boat

This is a story about what it takes to start a company. It may not seem like it, but it is.
I live on a boat in a harbour. It's an amazing place with just the right mix of artists, yacht owners and fishermen. As in all small communities there's a strong invisible pecking order. The fishermen are on top. They are the real men. They have survived force gale 12 storms in the bay of Biscay, they haul cod out of the freezing water with their bare hands in the winter, they move crates around the harbour like children moving toys across the lawn. And, of course they know all there is to know about boats. When it comes to boats they are the unquestionable authority. They have seen it all before, and they are always right. They are where you go for buying, selling, moving and fixing boats.

I like the fishermen. Without them I would have no idea how to fix all the things that inevitably break on my old boat. And I can tell they enjoy sharing their vast knowledge. But to them I'm not a real man. I'm an academic. I can't lift the heavy crates, I've never been in a real storm. And I can't haul cod, not even in the summer. To them I'm a boy.

posted by Max Tobiasen at 00:33

via maximise.dk

Nicely written story/parable about what it takes to do it.

Is /r/programming still /r/'programming'?

Tue, 30 Aug 2011 10:51:25 +0200

Today, I have to make a confession: I read reddit. Quite often, indeed. I do have my haggles with it, but for the time being, it's one of the best sources of fresh information there is.

My most frequented subreddit is /r/programming
I often see comments on this subreddit that say something to the effect of "this is not programming, go away to another subreddit!" (happened to the article below this one an hour ago).
The description of this subreddit on the right hand side is also pretty clear on the topic ("PROGRAMMING!"), basically saying that only posts with code are worthy programming posts.

However, the current top link is about an MITM attack, the second about VIM, then there are links about DOS attacks, encryption on paper (which to me sound like the opposite of programming), the UI of InfoQ and censorship. So there are quite a lot of posts that aren't specifically about programming as such, even though they are related with it and interesting for programmers, which is why they were submitted to /r/programming in the first place, I suppose.

Don't misunderstand me, I'm not complaining about the diversity of the links, in fact, I'm enjoying it.

The question I have is: Should we not embrace this diversity and relax our standards of "not programming"?

Defeating the MPAA with their own weapons

Tue, 30 Aug 2011 09:29:27 +0200

I was thinking a bit this morning about how one could use the MPAAs own tactics to defend against lawsuits by them. According to the DMCA (and similar directives in Europe), it is illegal to circumvent "technical protection measures" against copying and accessing copied material. Since this law applies to everyone, not only the consumers of protected material, let's do a little thought experiment:

Imagine that I go to a popular file sharing site and upload a file that contains a movie. Since I don't want anyone to sue me for doing that, I encrypt the movie using a standard software like GPG. This encryption should be so strong that not even the NSA can open my file and sue me (and if they do, we'd know a lot more about what the NSA knows about breaking GPG), so I should be safe from the movie studios, as the file I uploaded is in no way copyright protected as it stands. Now, of course I want my friends to be able to download and watch the movie, so I give them the key to the file, which could also be rather trivial. For example, the uploader's name, or the movie title, or the release date of the film or anything else.
I suppose I could even package the key and the encrypted file into an archive together. It wouldn't be much different from a BluRay disc, which contains a lot of software and some data that allows a suitably set-up device to decrypt the disc.
Well, but now I have added a "technical protection measure" and an access control to my file. Circumventing that measure, i.e. decrypting the disc when you are not the intended recipient would be illegal by the DMCA. Of course, I also added a file to the archive with a license stating that you can only use the file supplied for private use inside the state of Nigeria and only if you are the son of the overthrown dictator, but I guess most movie pirates wouldn't care about that.
As soon as I receive a suit for copyright infringement under the DMCA I can now apply that same right to me: Obviously the suing agency must have circumvented my copy protection measure and this be in breach of the DMCA. You could even argue that my encryption, which uses gold-standard GPG with a key length of 4096 bit is a lot more effective than the skimpy BluRay protection measures, so I should be entitled to a lot more damage compensation.

What do you think?

Functional Programming Is Hard,That's Why It's Good

Thu, 25 Aug 2011 19:40:00 +0200

Odds are, you don’t use a functional programming language every day. You probably aren’t getting paid to write code in Scala, Haskell, Erlang, F#, or a Lisp Dialect. The vast majority of people in the industry use OO languages like Python, Ruby, Java or C#–and they’re happy with them. Sure, they might occasionally use a “functional feature” like “blocks” now and then, but they aren’t writing functional code.

And yet, for years we’ve been told that functional languages are awesome. I still remember how confused I was when I first read ESR’s famous essay about learning Lisp. Most people are probably more familiar with Paul Graham’s “Beating The Averages” which makes the case that:

But with Lisp our development cycle was so fast that we could sometimes duplicate a new feature within a day or two of a competitor announcing it in a press release. By the time journalists covering the press release got round to calling us, we would have the new feature too.

A common thread among people proselytizing functional programming is that learning this new, functional language is “good for you”; almost like someone prescribing 30m in the gym a day will “make you fit,” but it also implies difficulty and dedication. Haskell, Ocaml, and Scala are different from Lisp in that they have a certain notoriety for being very hard to learn. Polite people call this “being broad & deep”. Less polite people call it “mental masturbation” or “academic wankery” or just plain “unnecessary.” I submit that this difficulty is a familiar situation, and it’s a strong indicator that learning one of these languages will make you more productive and competent at writing software.

Your First Time Wasn’t Gentle

...

via dave.fayr.am

Nice!

The Sagan Series (part 6) - End of an Era: The Final Shuttle Launch - YouTube

Sun, 21 Aug 2011 16:14:00 +0200

[youtube http://www.youtube.com/watch?v=3wJYpRJQVbo]

via youtube.com

Every time humanity stretches itself, it receives a jolt of productivity, that can carry for centuries.

Papers from Philosophical Transactions of the Royal Society, fro (download torrent) - TPB

Fri, 22 Jul 2011 08:28:00 +0200

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This archive contains 18,592 scientific publications totaling 33GiB, all from Philosophical Transactions of the Royal Society and which should be available to everyone at no cost, but most have previously only been made available at high prices through paywall gatekeepers like JSTOR. Limited access to the documents here is typically sold for $19USD per article, though some of the older ones are available as cheaply as $8. Purchasing access to this collection one article at a time would cost hundreds of thousands of dollars. Also included is the basic factual metadata allowing you to locate works by title, author, or publication date, and a checksum file to allow you to check for corruption.

ef8c02959e947d7f4e4699f399ade838431692d972661f145b782c2fa3ebcc6a sha256sum.txt

I've had these files for a long time, but I've been afraid that if I published them I would be subject to unjust legal harassment by those who profit from controlling access to these works. I now feel that I've been making the wrong decision. On July 19th 2011, Aaron Swartz was criminally charged by the US Attorney General's office for, effectively, downloading too many academic papers from JSTOR. Academic publishing is an odd system — the authors are not paid for their writing, nor are the peer reviewers (they're just more unpaid academics), and in some fields even the journal editors are unpaid. Sometimes the authors must even pay the publishers. And yet scientific publications are some of the most outrageously expensive pieces of literature you can buy. In the past, the high access fees supported the costly mechanical reproduction of niche paper journals, but online distribution has mostly made this function obsolete. As far as I can tell, the money paid for access today serves little significant purpose except to perpetuate dead business models.
The "publish or perish" pressure in academia gives the authors an impossibly weak negotiating position, and the existing system has enormous inertia. Those with the most power to change the system--the long-tenured luminary scholars whose works give legitimacy and prestige to the journals, ratherthan the other way around--are the least impacted by its failures. They are supported by institutions who invisibly provide access to all of the resources they need. And as the journals depend on them, they may ask for alterations to the standard contract without risking their career on the loss of a publication offer. Many don't even realize the extent to which academic work is inaccessible to the general public, nor do they realize what sort of work is being done outside universities that would benefit by it. Large publishers are now able to purchase the political clout needed to abuse the narrow commercial scope of copyright protection, extending it to completely inapplicable areas: slavish reproductions of historic documents and art, for example, and exploiting the labors of unpaid scientists. They're even able to make the taxpayers pay for their attacks on free society by pursuing criminal prosecution (copyright has classically been a civil matter) and by burdening public institutions with outrageous subscription fees. Copyright is a legal fiction representing a narrow compromise: we give up some of our natural right to exchange information in exchange for creating an economic incentive to author, so that we may all enjoy more works. When publishers abuse the system to prop up their existence, when they misrepresent the extent of copyright coverage, when they use threats of frivolous litigation to suppress the dissemination of publicly owned works, they are stealing from everyone else. Several years ago I came into possession, through rather boring and lawful means, of a large collection of JSTOR documents. These particular documents are the historic back archives of the Philosophical Transactions of the Royal Society — a prestigious scientific journal with a history extending back to the 1600s. The portion of the collection included in this archive, ones published prior to 1923 and therefore obviously in the public domain, total some 18,592 papers and 33 gigabytes of data. The documents are part of the shared heritage of all mankind, and are rightfully in the public domain, but they are not available freely. Instead the articles are available at $19 each--for one month's viewing, by one person, on one computer. It's a steal. From you. When I received these documents I had grand plans of uploading them to Wikipedia's sister site for reference works, Wikisource — where they could be tightly interlinked with Wikipedia, providing interesting historical context to the encyclopedia articles. For example, Uranus was discovered in 1781 by William Herschel; why not take a look at the paper where he originally disclosed his discovery? (Or one of the several follow on publications about its satellites, or the dozens of other papers he authored?) But I soon found the reality of the situation to be less than appealing: publishing the documents freely was likely to bring frivolous litigation from the publishers. As in many other cases, I could expect them to claim that their slavish reproduction — scanning the documents — created a new copyright interest. Or that distributing the documents complete with the trivial watermarks they added constituted unlawful copying of that mark. They might even pursue strawman criminal charges claiming that whoever obtained the files must have violated some kind of anti-hacking laws. In my discreet inquiry, I was unable to find anyone willing to cover the potentially unbounded legal costs I risked, even though the only unlawful action here is the fraudulent misuse of copyright by JSTOR and the Royal Society to withhold access from the public to that which is legally and morally everyone's property. In the meantime, and to great fanfare as part of their 350th anniversary,the RSOL opened up "free" access to their historic archives—but "free" only meant "with many odious terms", and access was limited to about 100 articles. All too often journals, galleries, and museums are becoming not disseminators of knowledge—as their lofty mission statements suggest—but censors of knowledge, because censoring is the one thing they do better than the Internet does. Stewardship and curation are valuable functions, but their value is negative when there is only one steward and one curator, whose judgment reigns supreme as the final word on what everyone else sees and knows. If their recommendations have value they can be heeded without the coercive abuse of copyright to silence competition. The liberal dissemination of knowledge is essential to scientific inquiry. More than in any other area, the application of restrictive copyright is inappropriate for academic works: there is no sticky question of how to pay authors or reviewers, as the publishers are already not paying them. And unlike 'mere' works of entertainment, liberal access to scientific work impacts the well-being of all mankind. Our continued survival may even depend on it. If I can remove even one dollar of ill-gained income from a poisonous industry which acts to suppress scientific and historic understanding, then whatever personal cost I suffer will be justified—it will be one less dollar spent in the war against knowledge. One less dollar spent lobbying for laws that make downloading too many scientific papers a crime. I had considered releasing this collection anonymously, but others pointed out that the obviously overzealous prosecutors of Aaron Swartz would probably accuse him of it and add it to their growing list of ridiculous charges. This didn't sit well with my conscience, and I generally believe that anything worth doing is worth attaching your name to. I'm interested in hearing about any enjoyable discoveries or even useful applications which come of this archive.
---- Greg Maxwell - July 20th 2011 gmaxwell@gmail.com
Bitcoin: 14csFEJHk3SYbkBmajyJ3ktpsd2TmwDEBb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAk4nlfwACgkQrIWTYrBBO/pK4QCfV/voN6IdZRU36Vy3xAedUMfzrJcAoNF4/QTdxYscvF2nklJdMzXFDwtF=
YlVR
-----END PGP SIGNATURE-----

It looks as if things are really going to get moving for the scientific community. This is another push for better, free access to scientific papers and journals in the form of 33GiB of papers which should be free. I wholeheartedly agree with everything Mr Maxwell writes, and if you care about the quality of science, so should you!

When Employees Misinterpret Managers // ben's blog

Thu, 21 Jul 2011 09:14:00 +0200

Time is money, so I went and bought a Rolex
—Wiz Khalifa, Phone Numbers

When I ran Opsware, we had the non-linear quarter problem also known affectionately as the hockey stick. The hockey stick refers to the shape of the revenue graph over the course of a quarter. Our hockey stick was so bad that one quarter, we booked 90% of our new bookings on the last day of the quarter. Sales patterns like this make it difficult to plan the business and are particularly harrowing when you are, as we were, a public company.

Naturally, I was determined to straighten out the hockey stick and bring some sanity to the business. I designed an incentive for sales people to close deals in the first two months of the quarter by issuing bonuses for deals in those months. As a result, the next quarter became slightly more linear, and slightly smaller than anticipated – deals just moved from the 3^rd month to the first two of the following quarter.

[...]

via bhorowitz.com

Really interesting comment on what to manage towards.

Shower

Wed, 20 Jul 2011 12:41:00 +0200

via pepelsbey.github.com

Shower is a great template for presentations in the web. Works on all important browsers and looks very neat.

Googles long-term memory

Sun, 19 Jun 2011 22:36:22 +0200

Recently, I made an interesting discovery regarding Googles long-term memory. I was shopping for some RAM modules for my notebook and I wanted to find out, which modules exactly I needed. So, I googled. I found the correct type, used a price-comparison website, ordered a few and that was that.

But, a few days later, this appeared on a page I was visiting:

A bit later, this one:

And then this:

And then these:

Now, that's funny. Obviously, after I had bought some memory modules, I was definitely in the market for some more. Or maybe a hard drive (which wouldn't even fit into the notebook). It looks like Google has an impressive long-term memory, storing my "interesting" searches for quite a while, because I only got them to stop by opting-out of ad-tracking completely. No, I find this a waste of resources, because there's a difference here between events that occur only once, like shopping for hardware, and events that occur more often, like shopping for books or such things. And I think it's strange that Google cannot differentiate between the two. After all, this should show up rather obviously in click statistics.

However, this is the first time such a strange campaign targeted myself. Have any of you seen such things?

Python Idioms and Efficiency Suggestions

Thu, 16 Jun 2011 09:37:00 +0200

Check out this website I found at jaynes.colorado.edu

Some very nice and very effective Python tips. Sweet!

Ask Slashdot: Best Certifications To Get? - Slashdot

Thu, 09 Jun 2011 08:03:00 +0200

A bit off topic, but you triggered something I've been thinking about for a couple of years. That "spark" is fluency.

I swtiched jobs from being a computer programmer to being an ESL teacher in Japan. Japan is somewhat famous for churning out students who know a lot *about* English, but can't order a drink at Mac Donald's. We used to have a name for those kinds of people with regard to programming languages: language laywers. They can answer any question you put to them *about* a programming language, but couldn't program to save their life. These people often make it past job interviews easily, but then turn out to be huge disappointments when they actually get down to work. I've read a lot about this problem, but the more I look at it, the more I realise that these disabled programmers are just like my students. They have a vocabulary of 5000 words, know every grammar rule in the book but just can't speak.

My current theory is that programming is quite literally writing. The vast majority of programming is not conceptually difficult (contrary to what a lot of people would have you believe). We only make it difficult because we suck at writing. The vast majority of programmers aren't fluent, and don't even have a desire to be fluent. They don't read other people's code. They don't recognise or use idioms. They don't think *in the programming language*. Most code sucks because we have the fluency equivalent of 3 year olds trying to write a novel. And so our programs are needlessly complex.

Those programmers with a "spark" are programmers who have an innate talent for the language. Or they are people who have read and read and read code. Or both. We teach programming wrong. We teach it the way Japanese teachers have been teaching English. We teach about programming and expect that students will spontaneously learn to write from this collection of facts.

In language acquisition there is a hypothesis called the "Input Hypothesis". It states that *all* language acquisition comes from "comprehensible input". That is, if you hear or read language that you can understand based on what you already know and from context, you will acquire it. Explanation does not help you acquire language. I believe the same is true of programming. We should be immersing students in good code. We should be burying them in idiom after idiom after idiom, allowing them to acquire the ability to program without explanation.

via ask.slashdot.org

Wonderful comparison between learning natural languages and learning programming languages.

I am scared

Tue, 24 May 2011 20:49:23 +0200

I guess it all started when I was a little kid. I didn't even notice, back then.

When I was about six, I learned about how the world has enough water for everyone, but unfortunately it's all salt water. I also learned about evaporation, and that you basically get distilled water for free with it. So, my little kids mind went to work combining these two effects and created a scheme to water the worlds deserts. It's quite simple, actually. You take a few sheets of glass, mount them at an angle, add a few pipes as well as a pump adding seawater under the panes, and instantly you can start harvesting freshwater suitable for nurturing crops. I was sure, with this simple scheme you could generate enough fresh water to make the deserts green. After all, that's what happens in clouds, too, right? And actually, that scheme does work: http://en.wikipedia.org/wiki/Solar_desalination

But, as you probably know, I did not succeed in my grandiose plans, and thus the deserts still remain unwatered. Why did I not do anything? Because I was scared. Scared by my own courage, scared by all the things no one else would do to help me, scared of leaving my cozy little middle-class comfort zone. So I remained inert, even though the little voice in my head mocked me for not doing what I knew was necessary.

Fifteen years later I finished school and went to university. A wonderful environment for anyone who wants to learn at crazy speeds, and so I did. I met lots of people, some of whom had started a business when they were still in school. I knew that this was what I wanted to do, too. Start a business, be productive, be successful, be happy. But there was so much else to do. Studying. Girls. Exams. Learning. And even though I read all the essays by Paul Graham, witnessed many Internet business started and ended, saw the bubble rise and burst and I was always this close to doing it myself. I could see the changes coming, could see all the niches still unfulfilled, some of them as large as whole industries were before. I felt the changes in people's behaviors when they found what the Internet and modern computers could do for them, and I knew that the technology would revolutionize most areas of business there are. I talked with my friends about how we would be part of that revolution, how we would kill the old beasts and be masters of the new economy. We had plans, we knew what we had to do, and in retrospect, we were actually right there, too. Most of the ideas were realized sooner or later by people who were scarily successful with them.
But I didn't do anything about it. I was scared. Scared to start without a degree, scared to be responsible for myself, scared to lose the fine little progress I had made for myself. I found arguments, easy and lazy arguments why I shouldn't start a business, why I should be careful and think about it some more. And yet, the little voice was still there, still mocking, still nagging.

It stayed that way for almost ten more years.

But then, somehow, the excuses started dropping away. I had a finished degree, I was safely employed, health-insured, in a stable relationship, had travelled far and wide, I had started doing sports, had bought all the things I never thought I'd buy and I had safely arrived in the adult world. I even started learning to play music to take my mind away from all the things I should be doing.

And then, finally, the last piece fell into place. I plunged. I quit my PhD studies, stayed employed half-time and started my first business. The voice in my head was excited: I was actually doing what I was supposed to do, and what so many had showed me before.

And now, I am two months in. I am more scared as ever, but I can see the things that scare me now, clearly. I'm still scared by all the red tape, of which I have to take care. I'm still scared of all the lawyers, accountants, guidelines, tax laws, chambers of commerce, customers, competitors and customers that are already involved in my business and who will be involved in the future. And I've only scratched the surface. But everything is so clear, now. There's no voice mocking me for my failure to launch, because I launched. There's no boss in my face to tell me what to do and what not to do, because I am my own boss. The scare is now in front of me, so I can cope with it.

It has taken twenty-five years, almost all my life, but I have finally arrived in the place I wanted to reach since I was a child. I took the muffled scares that surrounded me and placed them crystal clear in front of me. Now I can deal with them.

And so can you!

Dropbox is a huge torrent swarm

Tue, 24 May 2011 20:15:24 +0200

Funny, how the world turns, isn't it? Some weeks ago I wrote the following article. Then this happened: http://razorfast.com/2011/04/25/dropbox-attempts-to-kill-open-source-project/

And my article got relegated to the archive. I actually think it's quite neat, so here we go:

I recently found out something nice about Dropbox: When you put a file in your Dropbox that's already stored somewhere else, you don't have to upload it. It'll appear "magically" in your Dropbox folder, and if it's a shared folder will start syncing almost immediately at the other end.

In and of itself, this is nothing very surprising, Dropbox just computes a hash of the file (or probably some segments of that file) and saves itself the upload if it already knows the contents of the file.

The interesting thing is that this also works with files that aren't in your Dropbox, but in any Dropbox anywhere on the world. So, you put a file into your Dropbox folder, Dropbox does its hashing, finds the file somewhere else and puts a reference to the file into your Dropbox web folder.

Now, imagine what happens when you have two computers, both of them connected to Dropbox, and on different networks. You put a file into one of the Dropbox folders, presumably a large one. Dropbox finds the file for you without upload and starts streaming the file to your other computer instantly.

Interesting.

What if we could trick one of the Dropbox clients into thinking you had put a file with some specific hash into its folder? Exactly the same thing would happen. The other computer would start streaming that file instantly from Dropbox's magic file cloud without you ever having a copy.

Note: I am not the first person to think of this:

http://stackoverflow.com/questions/4767505/exploit-dropbox-file-redundancy-check

Now, where do we get the necessary data for those files? As it so happens, we already have a system that is based on exactly that: transmitting file hashes and file chunk hashes, namely BitTorrent sites. What we could do now is to simply take these hashes, fix up a "zombie" Dropbox client that accepts these hashes instead of files and have any of the many torrented files streamed to you instantly via the Dropbox servers.

How lucky is it for Dropbox, then, that the two systems don't fit together exactly...

http://stackoverflow.com/questions/1903416/do-any-common-os-file-systems-use-hashes-to-avoid-storing-the-same-content-data-m

Readable / TastefulWords.com

Mon, 09 May 2011 16:51:00 +0200

Whenever you're on an article page, no matter the website, just click the Readable bookmarklet; a reformatted version of the article will appear on top of the original page.

"What's an article page?", you ask; good question. An article page is any page that contains one large block of text — like, for example, all newspaper articles. Readable is only designed to automatically process these kinds of pages.

On any other type of page, you can still take advantage of Readable, but in manual mode: simply select the text you would like Readable to display, and then click the bookmarklet.

When you're done reading, double-click the background to dismiss the Readable overlay.

via readable.tastefulwords.com

Readable -- what Readability was before it became something else.

The better iTunes

Sun, 01 May 2011 10:20:39 +0200

I recently discovered Clementine ( http://www.clementine-player.org/ ). It's what Amarok and iTunes started out as and what they should still be: music players. It organizes your collection, copies files where you want them, lets you search and catalog your music and it plays it. And it's cross-platform.

Nothing more, nothing less.

And bonus points for Fredrika Stahl on the front page.

The T-Mobile Royal Wedding

Sun, 01 May 2011 10:17:00 +0200

via youtube.com

It, really, IS the better royal wedding!

Shapeways | blog: Theo Jansen's 3D Printed Strandbeests

Sat, 30 Apr 2011 12:18:00 +0200

[youtube http://www.youtube.com/watch?v=XjDktMn3gA4&fs=1&hl=nl_NL]

via shapeways.com

Finally, someone is putting those 3D-printers to good use...

WTF! Insane Human Curling by BIC Flex 3

Sun, 17 Apr 2011 13:58:00 +0200

via youtube.com

What the heck?

the walrus and the oyster: Life Lessons

Mon, 11 Apr 2011 18:07:00 +0200

http://www.thewalrusandtheoyster.com/2011/04/life-lessons.html

Six very important life lessons.

Access Main Computer File

Mon, 11 Apr 2011 17:57:00 +0200

via accessmaincomputerfile.net

History of computer GUIs in film.

Tune for Two (2011) on Vimeo

Mon, 11 Apr 2011 17:51:00 +0200

[vimeo http://www.vimeo.com/21362582 w=500&h=283]

via vimeo.com

These are dark times, when you need to have a little fun!

Language of the month: Dead in the Water

Sat, 26 Mar 2011 19:56:00 +0100

I was just reading an article called "Language of the Month: Mirah"

about a programming language called 'Mirah' on the JVM. It's a really nicely written article, the language seems good enough and all the points in the article are well thought-out and well articulated.

However, I stopped reading when I came to this bit:

puts "Hello, world!"
We have our first program in Mirah. Let's run it with the mirahcommand:

(from http://drdobbs.com/java/229400307#)

Mirah will never catch on, and you can see that fact from this simple line. And half of the intended and clearly-stated target audience already browsed on. Why? Because it puts, instead of printing. It looks and sounds as if that is such a simple thing, but it's actually something a lot deeper.

No one cares about the difference between puts and prints when you want to know how to output something to the screen. In most of the software we write, that never happens anyway, because we have graphical interfaces, and HTTP, and logging facades. However, the language designers chose to use a different keyword for one of the most basic interactions with the language than what most users, coming from Java, would be used to, or would guess on the first try. They probably had very good reasons to do so, but it just turns away so many people trying out your new language and it probably prevents you from reaching critical mass.

Also, this little fact shows that you care more about the functional background and history of your language than about mass-scale adoption. And in fact, the next highlight is only a few lines away:

public static String hello(String name) {  return "Hello, " + name + "!";}

def hello(name:String)  "Hello, #{name}!"end

Ok, the syntax doesn't look too bad, to my exotic-language-trained eyes. But to a Java programmer? Why does it not state the return type? Why are name and type switched around in the parameter list? And where the heck do they define their blocks? Mixed indentation and end-token syntax? What? Why? The next 30% of your audience is gone at this point.

It goes on and on and on:

def run    frame = JFrame.new @title    frame.setSize @width, @height    button = JButton.new "Press me"    frame.add button    button.addActionListener do |event|      JButton(event.getSource).setText "Mirah rocks!"    end    frame.setVisible true  end

Yeah, I see that that's Ruby. But why not use Ruby, then? It's not even worth going through the rest of the article, because there's only a fraction of the audience still remaining, and those are nodding their heads, saying to themselves: "Looks nice, like Ruby. Oh, I have to try Ruby again sometime." It's really a pity, because the language looks nice enough and the productivity you'd get out of it is probably five times that of Java.

But, Mirah is never going to be a mainstream language. Because it puts out.

(Yeah, I saved a great pun for last...)

MicroISV on a Shoestring: B2C stands for "Bingo To Customer"

Sat, 26 Mar 2011 19:09:00 +0100

via kalzumeus.com

Wonderful, and so true. Makes me want to try a million different things right now.

Mercurial RSS

Sun, 20 Mar 2011 18:59:00 +0100

The RSS feed that is built into Mercurial is a great thing. It let's you track easily who's checking in and what get's done in your repositories.

The preconfigured layout, however, is a bit spartan. It only shows the date, the commit message and the author. This always bugged me, because I'd like to see how much was done and how many files were changed on each commit.

Luckily, the contents of the RSS feed get prepared through a template engine. This makes it possible to change the contents of the feed without needing to muck around with too much stuff.

The following repeats much of the information you can also find on http://mercurial.selenic.com/wiki/Theming but is specific to RSS feed customization.

If you're on Ubuntu, the files we're going to look at are stored in

/usr/share/mercurial/templates/rss/

The feeds use the same framework as the web-served styles, which is why the layout and the file structures are a bit confusing at first. Basically, the RSS feed is composed of three files: the header (header.tmpl), the actual content (changelog.tmpl), within which single items (changelogentry.tmpl) are looped, and the footer (footer.htmpl). Since RSS feeds don't offer much in terms of customizability, we can focus on each individual entry, stored in changelogentry.tmpl. Looking at that give us this:

<item> <title>{desc|strip|firstline|strip|escape}</title> <guid isPermaLink="true">{urlbase}{url}rev/{node|short}</guid> <description><![CDATA[{desc|strip|escape|addbreaks|nonempty}]]></description> <author>{author|obfuscate}</author> <pubDate>{date|rfc822date}</pubDate> </item>

Looks nice. But I don't really need the description twice. So let's change the contents of <description> (which is what get's displayed in the RSS reader as the contents) with a list of changed files. For that, we simply exchange the description line, so that the file looks like this:

<item> <title>{desc|strip|firstline|strip|escape}</title> <guid isPermaLink="true">{urlbase}{url}rev/{node|short}</guid> <description><![CDATA[{files}]]></description> <author>{author|obfuscate}</author> <pubDate>{date|rfc822date}</pubDate> </item>

Trying this, however, results an error. The reason for this is because the template engine will give us a "filedifflink" for each file, i.e. we have to prepare a template fragment for the filedifflink. The template fragments are stored in the file ''map'' and to define the fragment, we simply add this line to the file:

filedifflink = '{file|escape}<br/>'

And that's all. Now the RSS feed contains for each entry a list of files that were changed, complete with a link to the file changes on the web frontend.

Muffin - Wikipedia, the free encyclopedia

Sat, 19 Mar 2011 12:22:00 +0100

The corn muffin is the official state muffin of Massachusetts.^[5]
The blueberry muffin is the official state muffin of Minnesota.^[6]

via en.wikipedia.org

And I didn't even know states had an "official muffin".

OK Go - End Love

Sat, 19 Mar 2011 11:43:00 +0100

via youtube.com

Wonderful new video by OK go.

Germany's new boom: making money by making stuff | World news | The Guardian

Wed, 16 Mar 2011 15:36:00 +0100

via guardian.co.uk

Looks like good old Europe did do something right after all. Amazing!

Frenzy - The Dropbox powered social network

Wed, 16 Mar 2011 14:07:00 +0100

via frenzyapp.com

That is one brilliant idea! I love it, and I will try it out.

Lego Carcassonne

Mon, 07 Mar 2011 11:28:00 +0100

via iamcal.com

Clever trick and nice build. You could extend it to a building session, where everyone can build up real cities into their parts.

The Impersonal Non-Computer

Sun, 06 Mar 2011 10:56:00 +0100

A few days ago, the iPad 2 was unveiled by Steve Jobs, and already people are gushing over how much better the world is going to be now that we have a faster iPad.

One piece I have read with much interest is Daring Fireballs "The Chair". It is well written, not too enthusiastic and he makes a few good points. However, there is something I have a lot of problems with, and it's the last sentence:

It’s a shame, almost, that we squandered the term “personal computer” 30 years ago.

Now, while this is a very good ending sentence for the article, I disagree with almost every point of the sentence. First off is the implication that the term "personal computer" was squandered on the distinction between mainframes, where you timeshared the computer and single-user computers. This distinction was a very important one, as it started the trend towards each and everyone having one or more computers, as well as towards miniaturization, as opposed to room-filling mainframe servers. In this way, computers have become more and more personal over the years, indeed, going so far that I can now buy fully assembled and working computers with a complete selection of software installed to my tastes in almost any electronics store. In fact, the computers that I am working on have exactly the software and exactly the settings I need and I want, sometimes frustrating my friends and coworkers to the extent that they'd rather user their own computer than mine. I suppose that the term "personal computer" is very fitting for that device, and I don't think the name was "squandered" on that kind of machine.

But the second gripe I have is with the implication that the iPad should actually be a personal computer at all. Apart from the fact that it doesn't fit the generally accepted cluster of things we call "PC"s very well, it is not very personal at all. All the iPads I have seen have had the same look, the same background picture, the same interaction modes, almost the same software installed and were used for almost exactly the same things (namely, Angry Birds and Fruit Ninja). Now, that is not a bad thing, and actually, what most people want and need are monocultures and pre-chewed content, as is apparent from the success that mainstream tv and music have had, and that Apple now has. But it can't really be called "personal" in any meaningful way, not much more than the big tv stations can be called "personal" for reading out twitter messages.

And finally, for me and for the most people, the iPad is not actually a computer. For me, having a computer science background, computers are inherently programmable devices. And I cannot program my iPad in any interesting way. Oh, sure, you say, I could spend a hundred dollars a year for the developer license, another 1500 on a Macbook and then the 500 for the iPad itself, and for a mere 2000 dollars I can get a programmable computer, for some time. But you know what? For 2000 dollars, I can get together with a few of my friends and let them do computations for me. I wouldn't call a room full of my friends doing computations a computer, even though they're a lot more personal to me than the iPad is.

Now, I covet Apple products as much as the next guy, and the iPad is a beautiful device and I'd like everyone to have one. I just don't think we should epitomize it as the One True Computer. And while iPads are bringing computing devices to more and more people, we should certainly not compare the introduction of a toy to the massive, worldwide revolution that was brought on by personal computing.

Jenga Driven Development

Wed, 02 Mar 2011 15:05:00 +0100

via programmingzen.com

I have worked with legacy projects and the picture explains exactly how it feels...

Trent Walton

Sun, 20 Feb 2011 10:41:00 +0100

Check out this website I found at trentwalton.com

Very well worth the read, as it hits exactly what I feel about how work should be. If you're not into the piece of work you're doing right now, you should complete it and then find something that offsets what you missed that time.

The Best Questions For A First Date « OkTrends

Sun, 20 Feb 2011 09:40:00 +0100

Okay, if you want to know...

Will my date have sex on the first date?

Ask...

Do you like the taste of beer?

via blog.okcupid.com

Hilarious statistical correlation analysis of the best questions to ask on a first date and the deeper issues you can find out about. However, not very relevant for us Europeans, because all of the questions asked have very different contexts in Europe -- especially the beer question.

Flight of the Bumblebee on 101 Bottles!

Fri, 18 Feb 2011 22:43:00 +0100

via youtube.com

Wonderful idea for after big parties.

Mark Hughes's answer to Given our current technology and with the proper training, would it be possible for someone to become Batman? - Quora

Fri, 18 Feb 2011 19:28:00 +0100

Anyway, there you are, rooftop to rooftop, and it occurs to you that the cop cars are so far gone now that you barely hear the sirens. So you think "Hmm, no wonder the real Batman has a car, this rooftop thing looks cool but I'll never make it in time to stop a crime that isn't happening within a block or two.

via quora.com

Wonderful and eloquent analysis of how you are going to be Batman.

Even though it totally kills the fantasy...

journal.stuffwithstuff.com » Blog Archive » Extending Syntax from Within a Language

Mon, 14 Feb 2011 19:29:00 +0100

A big goal with my little language Magpie is to do as much as possible at the library level and keep the core language small. I’ve been toiling for the past few weeks and I finally moved a huge chunk of Magpie over to the library side: all infix operators are now defined and implemented in Magpie. As far as the core language is concerned, it doesn’t even have operators.

Not only can you now define your own infix operators, including specifying precedence and associativity, but other large chunks of Magpie syntax are now defined at the library level.

via journal.stuffwithstuff.com

Wonderful, simply wonderful. I encourage you to read the whole article and I encourage myself to try Magpie soon...

How to Get a Raise | SebastianMarshall.com: Strategy, Philosophy, Self-Discipline, Science. Victory.

Sun, 13 Feb 2011 21:35:00 +0100

The question was, “Why do business analysts and project managers get higher salaries than programmers?”

My reply -

Because programmers don’t ask for more money often enough.

via sebastianmarshall.com

He's right. Not all of what he says afterwards is completely true for salaried employees everywhere, but the basic idea is correct.

What I find fascinating as well is that for a business analyst or a customer-facing consultant, asking for more money is part of the job. So they get a lot of training talking to people and asking for more, while the technical, background-loving people don't.

So, let's all start practising and asking for raises. If you don't get one -- no harm done, you can always go on doing what you already do...

Graham Wetzler: Why Dropbox Succeeded and Syncplicity Didn’t

Wed, 02 Feb 2011 20:21:00 +0100

Check out this website I found at wetzler.me

It's a strange thing, noting that some service was "better" because it has less features. I have to try that sometimes.

ReMarkdown: HTML as Markdown text using CSS

Thu, 27 Jan 2011 12:34:00 +0100

Check out this website I found at covertprestige.info

Remarkdown -- lets you convert HTML to Markdown. Nice hack.

while true blog » What if Visual Studio had Achievements?

Wed, 26 Jan 2011 21:14:00 +0100

What if Visual Studio supported achievements, just like games on Steam, Xbox or PS3? Bragging to your coworkers about which one you’ve just unlocked, imagine that! Here’s a little proposed list for some of them. .NET / C# flavored, of course.

Falling Down – Created a new SharePoint project

Job Security – Written a LINQ query with over 30 lines of code

The Sword Fighter – 5 Consecutive Solution Rebuilds with zero code changes

Shotgun Debugging – 5 Consecutive Solution Rebuilds with a single character change

The Mathematician – Defined 15 local variables with a single character name

The Academic – Written 1000 lines of F#

Spaghetti Monster – Written a single line with more than 300 characters

Wild One – Mixed tabs and spaces for indentation more than 5 times in a single line

The Organizer – Created a Solution with more than 50 projects

The Portal – Created a circular project dependency

The Multitasker – Have more than 50 source files open at the same time

The Code Keeper – Uninstalled Resharper because it made you redundant

Pasta Chef – Created a class with more than 100 fields, properties or methods

Procedural Programmer – Created a method with more than 10 out parameters

Steam Powered – Added Visual Studio as a Steam game

The Poet – Written a source file with more than 10,000 lines

The Enterprise – Build Solution took more than 10 minutes

Highway to Hell – Successfully created a WCF service

The Explainer – Written a comment with more than 100 words

TPS Reports – Created a Crystal Reports Project

Rage Quit – ALT+F4 after a failed bug fix

Ooooh Shiny – Written 100 extensions methods

Look Ma – Written an infinite Fibonacci generator using yield

The Engineer – Killed a zombie with The Wrench

The Architect – Created 25 Interfaces in a single project

The Right Way – Test method is longer than the tested method

The Defender – Checked every argument for null exceptions

Pokemon Programming – Caught all the exceptions

Black Magic – Implemented a RealProxy

Gimme back my ASM – Used ILGenerator

I’m Sorry – Created a new Visual Basic Project

The SEO Expert – ASP.NET MVC Routing table with more than 100 routes

The Matrix – Windows Forms with more than 100 controls

The Daredevil – UpdatePanels nested more than 3 layers deep

Just a Test – Nested multiline C-style comments that caused a compilation error

Warm Bath – Successfully consumed a non .NET SOAP web service

Old School – Defined more than 100 static objects

The Cloner – Copy-pasted more than 50 lines

The Dependency – Referenced more than 30 projects

Paying the bills – Imported a Visual Basic project

First Hit – Included a Codeproject.com library into your project and it actually compiled

Paula – Define a firstname field with value Brillant

Every Option Considered – Created an enum with more than 30 values

Inspired by Steam Holiday sales and Battlefield Bad Company 2. Odd web coding exposed on the most minimalistic company page possible. Enjoy.

via blog.whiletrue.com

Good question. And how can we make it happen?

ILUVUXDESIGN on Vimeo

Wed, 26 Jan 2011 21:12:00 +0100

via vimeo.com

The UX Designer -- a mythical creature...

Saturn's hexagon recreated in the laboratory - The Planetary Society Blog | The Planetary Society

Wed, 26 Jan 2011 21:12:00 +0100

via planetary.org

The Saturn Hexagon deciphered.

Development Chaos Theory » Blog Archive » How (not) to write Factorial in Java.

Wed, 26 Jan 2011 21:10:00 +0100

I will admit this post was inspired by How would you write factorial(n) in java? So excuse me while I rant in code: I have a point to make at the bottom of this article.

Now most people who write factorial in Java may start with something simple, like:
public static int factorial(int n) {
    if (n == 0) return 1;
    return n * factorial(n-1);
}
Wrapping this in a class (since we’re talking about Java, of course), they may write it inside a utility class. For example:
public class FactorialUtil {
    public static int factorial(int n) {
        if (n == 0) return 1;
        return n * factorial(n-1);
    }
}
Simple, no?

There is a non-recursive solution as well:
public class FactorialUtil {
    public static int factorial(int n) {
        int ret = 1;
        for (int i = 1; i <= n; ++i) ret *= i;
        return ret;
    }
}
Now the observent reader may note “gosh, it’s entirely possible the value will be longer than an integer”, so they’ll perhaps rewrite in terms of BigInteger or at least in terms of long, depending on the requirements of the program. Thus:

...

via chaosinmotion.com

Very funny. And, unfortunately, very true.

How I Screwed Yasser Arafat out of $2mm (and lost $100mm in the process) -- by Altucher Confidential

Tue, 25 Jan 2011 08:54:00 +0100

via jamesaltucher.com

Nice story.

iTerm 2

Sun, 23 Jan 2011 18:09:00 +0100

What is iTerm2?
iTerm2 is a replacement for Terminal and the successor to iTerm. It works on Macs with Leopard or Snow Leopard. Its focus is on performance, internationalization, and supporting innovative features that make your life better.

Why Do I Want It?

Compare iTerm2 to iTerm, or compare iTerm2 to Terminal.app. Still unsure? Check out the features and screenshots!

How Do I Use It?

Try the FAQ or the docs. Got problems or ideas? Report them in the bug tracker, or take it to the forum. Or tweet to @gnachman and I'll do my best to respond.

via sites.google.com

I just found iTerm, an open source replacement of the standard Mac Terminal.app. Looks great, even though it's still in alpha stage.

A brief glimpse of Nokia's popularity outside the Western world

Thu, 20 Jan 2011 23:03:00 +0100

via downloadsquad.switched.com

Another look at our predisposed views, especially regarding telephones. Shown here is the number of chinese mobile phone users visiting different app stores. Note that both the Android and the Windows app store are in front of Apple, and all three together make up about half of the Nokia OVI Store.

There needs to be a Twitter Domain Name Service

Tue, 18 Jan 2011 22:30:00 +0100

I recently acquired a domain name in preparation for a product launch shortly. Being the cool and suave marketing expert that I am, two weeks later I have the idea to get the matching twitter account. Alas, the name is taken and, to make matters worse, by some guy whose only tweet every, was "schooll...", two years ago. So, in the end, I'll have to settle for example_com, or perhaps example'com, since example.com is not an allowed twitter name.

What would be really cool, though, would be if Twitter allowed a period in their twitter names - but only for domain name owners. Verification is easy, too, just put

<div class="tweetdomains"><div class="email">youremail@example.com</div></div>

on your index page, go to twitter.com/tweetdomain and get your password sent to you.

Twitter: wouldn't you love 200 million instantaneous new accounts?

P.S.: Yes, I know, there are technical challenges, especially when altering a data scheme like proposed here for the names. But hey, no pain, no gain!

The Nokia N9 Blog » Blog Archive » The world’s most revolutionary mobile phone

Tue, 18 Jan 2011 21:17:39 +0100

Sometimes I get the wonderful opportunity to reevaluate how I think about things. This time it happened about influence:

http://thenokian9blog.com/2011/01/05/the-worlds-most-revolutionary-mobile-phone/

Of course I was of the opinion that the iPhone must have been the most influential phone in recent years, because from my perspective, it changed the way we interacted with the internet and with phones in general, which was an important change. Much more important, however, was the change for the millions upon millions of people who, for the first time in history, did have access to the telephone network at all. Which, I guess, is more important than twittering about your latest meal...

Things Real People Don't Say About Advertising

Sun, 16 Jan 2011 20:09:00 +0100

via tpdsaa.tumblr.com

Things don't say about advertising: Saying things about advertising, no one else has the guts to.

lonelysandwich

Sun, 16 Jan 2011 20:07:00 +0100

3D Without Glasses, With Nightmarish Disfigurement

We’ve all seen red-cyan (anaglyph) glasses for 3D viewing. And of course, the more current polarization (RealD Cinema) and even the high-tech Liquid Crystal Shutter glasses. Now, from this guy, comes the patented Oh My Christ Turn it Off Turn it Off Goddamn It You’re Scaring the Children What is Wrong with Youvision™.

via neonmarg

via lonelysandwich.com

Wonderful parody of the recent 3d-craze.

Andy Swan » “OK, but there are two rules…”

Sun, 16 Jan 2011 20:04:00 +0100

“OK, but there are two rules…”

January 5, 2011 – 12:35 pm

Until recently, Makers Mark has had exactly one product on the shelves since its introduction in 1959. Their response when asked why only one product? ”This is the best bourbon.” But even consumers who agreed demanded variety…something different every once in a while.

The Master Distiller agreed, and persuaded the president, Bill Samuels, Jr,. to at least let him TRY to create a new premium product.

Samuels response:

“OK, but there are two rules: 1. You can’t change the recipe. 2. You can’t do anything that anyone else has done before.”

With those two options off the table, the only alternative was to invent something completely new, or do nothing at all. Long story short….the distiller got to thinking, invented an entirely new process at the end of the aging cycle, and now they have a new product, Makers Mark 46.

The buzz in the bourbon world was huge. The product is a hit.

What Bill Samuels did was force innovation through voluntary restriction. He specifically restricted his master distiller from doing THE two things that every other company in the world would have done: Change the recipe slightly or create a premium “single barrel” or longer-aged version.

By not allowing yourself to use all of the tools at your disposal….by not allowing yourself to do the things that normal people do….by not allowing yourself to compete on the features that every other product has….you can force yourself to innovate, differentiate, and create a unique value proposition.

Running a real estate website? OK….you’re not allowed to show the asking price or address of any home. Go.

Starting a pizza place? OK….no tables or chairs allowed.

New app to compete with runkeeper? No user inputs allowed. None. Just figure out when they are running.

Kick the chair out from under you and I bet you’ll stand taller than ever before (or at least fail faster and try something else!)

Force yourself to innovate through voluntary restriction.

Win.

via andyswan.com

One way to make innovation happen: Add some useful but artificial constraints. Not only does that make the newly-created product special, but it also makes you think about what you're trying to achieve.

If You Are So Smart, Why Aren't You an Entrepreneur? Returns to Cognitive and Social Ability: Entrepreneurs versus Employees

Sun, 16 Jan 2011 19:55:00 +0100

Found something interesting at http://ideas.repec.org/p/iza/izadps/dp3648.html

The smarter you are, the more money you earn -- so far not very surprising. The interesting result, however, is that this works better for entrepreneurs than for employees.

So, in effect, if you are smart, you should be an entrepreneur.

I wonder how much of that effect can be explained by selection, both self-selection and external selection by bankruptcy.

NASA - The Frontier Is Everywhere

Tue, 11 Jan 2011 14:31:00 +0100

via youtube.com

Wonderful.

There Was Once a Woman Who Had Immortal Cells

Wed, 29 Dec 2010 15:56:00 +0100

via todayifoundout.com

A bit on the popular side, but very interesting.

Agile Plumbers

Sat, 25 Dec 2010 15:30:00 +0100

Check out this website I found at david.ing.name

Nice. Didn't know there was a special tag for Posterous-content. Anyway, nice satire on how Agile would work in the real world.

Fast Iterations: How Napoleon Managed the French Army - Managing Metrics

Sat, 11 Dec 2010 15:11:00 +0100

The armies of the late 1700’s would have looked familiar to Julius Caesar. Thousands of men marching down roads in thin columns with hundreds of supply wagons following behind. The wagons were unable to move across the muddy roads and often forced all the troops to stop and camp whenever the weather turned bad. Opposing armies lumbered slowly across the countryside until they eventually met at the field of battle. The soldiers fanned out into lines that were two men deep with one line firing while the other reloaded. The supply wagons were positioned behind the attacking lines and shuttled food and ammunition up to the fighting and the wounded back to be cared for.
The tactics of war also remained unchanged since Roman times. Generals commanded the forces from the back of the battlefield with scouts in the front closely observing enemy movements. The scouts would carry information back to the generals, who would make decisions and send their orders back to the front.

via managingmetrics.com

Agile management styles are nothing new. Even Napoleon used them

Apple engineer re-creates ancient computer with Legos

Sat, 11 Dec 2010 15:09:00 +0100

via news.cnet.com

Wonderful. Really wonderful. And very well-produced video.

Curved White, Architecture

Mon, 29 Nov 2010 19:45:00 +0100

via curvedwhite.com

Beautiful!

Are You a Manager or a Leader? Who Cares! Just Do What’s Needed! — MakingITclear®

Mon, 29 Nov 2010 19:05:00 +0100

Business literature is full of distinctions that some very smart people make between a manager and a leader:

“Management is doing things right; leadership is doing the right things.” – Peter Drucker

“…Leaders are concerned with what things mean to people. Managers are concerned about how things get done.” – Abraham Zaleznik

[...]

But the bottom line is that the distinction between a manager and a leader is all semantics. No one hires you with the job title of Manager and then says, “Now don’t try leading — that’s not in your job description.” And no one hires you with the job title of Leader and then says, “Just focus on leading — managing isn’t something you should care about.” Let’s face it: All management jobs have a leadership component, and all leadership jobs have a management component. And frankly, it doesn’t really matter what you call it; what matters is that you get the job done.

via blog.makingitclear.com

Very true indeed...

American Software: Lovely Video

Sat, 20 Nov 2010 13:30:33 +0100

A wonderful love story and very well done.

(from https://vimeo.com/15947949)

Link: A New Era for Design

Sat, 20 Nov 2010 12:35:25 +0100

https://www.kickstarter.com/blog/a-new-era-for-design

Very interesting essay on the future of design.