I remember reading this back in 2015, but how is this not just the age old conservative (don't do more than needed) vs progressives (lets try some new risk things) debate? As much as I can relate to a more conservative choice when choosing tech that might power a giant consumer company, sometimes using riskier tech in a startup makes more sense to get the real innovation flowing...
and the idea of "Choose New Technology, Sometimes" just feels like a little cheat to get away from the bigger issue with the overall thesis. In this way, the idea in this blog just feels so out of touch.
It’s not about conservative vs progressive but about “safe, battle-tested, stable” vs “could be amazing future-tech”; and the idea of having a small number of innovative technologies, in particular where they matter, is about allowing yourself to spend all those thought-tokens on areas that matter.
You don’t want to be innovating your logging tech or your database if the old will work because you might spend 90% of your time trying to get your new, creative logging solution working when you should be spending it on the creative part of your business.
I might be missing something but conceptually, “safe, battle-tested, stable” vs “could be amazing future-tech” IS about conservative vs progressive... am I missing something?
Good point, I suppose conservative is the right word there. Its broader meaning has been a bit louder lately, so I didn’t think of it the right way there.
I feel like there's a common theme in business advice like this, the bullshit jobs thesis, general alienation of workers, that individuals have a lot of tunnel vision and don't understand the place of a firm in capitalism. The reality is we don't know what strategies will and won't succeed. The system overall works to the extent it works because a hundred thousand monkeys throwing shit at the wall are bound to hit a few targets at least a few times, even if the vast majority of them miss. The larger economy and investment ecosystems are set up under the assumption that most products will never see a single user and most companies will fail. Most employees will spend much of their careers working on projects that get scrapped. It's not much different than life itself. Evolution has no master plan. Most organisms never reproduce. Most species barely make a blip in any ecosystem before going extinct. It sucks for the individuals, but the larger collective goes on and thrives. Life itself does fine. The economy at large does fine, in spite of the ebbs and flows of the business cycle with its bubbles and recessions. Everything still works out in the long run. It's not only okay to fail but it's crucial to the long run success of our collective endeavors.
The vast majority of business strategy is just the lucky survivors assuming in retrospect they were smarter than everyone else. I've been seeing it my entire life, not just in business but in the military, in politics, in sports, in entertainment. Some project leader has one success and coasts off of that forever, failing over and over again afterward because they keep getting rehired, reelected, or re-invested in by inherently conservative investors, hiring committees, team owners, voters, party leaders, who blindly assume winning a Super Bowl means you must have known what you were doing, no matter how many random bounces going your way you had no control over it took.
This is unrelated to this article, but I see such simple titles posted on HN often and given how many articles I read per day on HN, I don't know if it's worth me reading or not until I click it. I wish we had a feature on HN that semantically defined who the intended audience for an article is, specially for such opaque titles. Something like the following (used gemini for this):
Here are the 1-2 tags defining the intended audience for each article on the front page:
Five frontier LLMs disagree on 67% of 1k real-world fact-check claims
Tags: AI Researchers, Machine Learning Engineers
YouTube to automatically label AI-generated videos
Tags: Digital Content Creators, General Tech Consumers
A Eureka machine that thinks like nature and explores what AI cannot
Tags: Computer Scientists, AI Researchers
AMD pulls a bait-and-switch on Linux users with Vivado licensing changes
Tags: Linux Users, Hardware Engineers
I analysed 20 years of my chats
Tags: Data Enthusiasts, Hobbyist Programmers
I think Anthropic and OpenAI have found product-market fit
Tags: Tech Entrepreneurs, Product Managers
Super confusing... seems like some sort of in with the VCs that can pull this program's guests was enough to create a new podcast that is now seen as influential. My best is, this was a side liquidity event for the openAI VCs that had somehow invested into the podcast, looking to get some money out of openAI stake.
> It's only true in a universe where Iran would have collapsed from within before the expiration of the sunset clause, and that clearly was not going to happen.
No one can know this hypothetical, but some def bet their entire futures/careers on this: that an Iran with a more prosperous middle class (as a result of JCPOA) might have had a better chance for social/internal reform, i.e. regime change.
> If you take a lot of chances, that adds up eventually and you'll have some big wins. Just do it safely, so that they don't add up to a lot of big losses, too.
And here is great contradiction in this whole essay. You can't "safely" take a lot of chances and not lose big, when in most cases to have big wins, one has to do unsafe things...
This is also why folks who have a safety net (in terms of family wealth, etc) tend to do better as entrepreneurs. Not sure this essay is helpful.
Only to prompt thought on this exact question, im interested in answers:
I just ran a benchmark against haiku of a very simple document classification task that at the moment we farm out to haiku in parallel. very naive same prompt system via same api AWS bedrock, and can see that the a few of the 4b models are pretty good match, and could be easily run locally or just for cheap via a hosted provider. The "how much data and how much improvement" is a question i dont have a good intuition for anymore. I dont even have an order of magnitude guess on those two axis.
Heres raw numbers to spark discussion:
| Model | DocType% | Year% | Subject% | In $/MTok |
percents are doc type (categorical), year, and subject name match against haiku. just uses the first 4 pages.
in the old world where these were my own in house models, id be interested in seeing if i could uplift those nubmers with traingin, but i haven't done that with the new LLMs in a while. keen to get even a finger to the air if possible.
Can easily generate tens of thousands of examples.
You can fine tune a small LLM with a few thousand examples in just a few hours for a few dollars. It can be a bit tricky to host, but if you share a rough idea of the volume and whether this needs to be real-time or batched, I could list some of the tradeoffs you'd think about.
Source: Consulted for a few companies to help them finetune a bunch of LLMs. Typical categorical / data extraction use cases would have ~10x fewer errors at 100x lower inference cost than using the OpenAI models at the time.
ok, even that "few thousand examples" heuristic is useful. the usecase would be to run this task over id say somewhere in the order of magnitude of 100k extractions in a run, batched not real time, and we'd be interested in (and already do) reruns regularly with minor tweaks to the extracted blob (1-10 simple fields, nothing complex).
My interest in fine tuning at all is based on an adjacent interest in self hosting small models, although i tested this on aws bedrock for ease of comparison, so my hope is that given we are self hosting, then fine tuning and hosting our tuned model shouldn't be terribly difficult, at least compared to managed finetuning solutions on cloud providers which im generally wary of. Happy for those assumptions to be challenged.
Labeling or categorization tasks like this are the bread and butter of small fine tuned models. Especially if you need outputs in a specific json format or whatever.
I did an experiment where I did very simple SFT on Mistral 7b and it was extremely good at converting receipt images into structured json outputs and I only used 1,000 examples. The difficulty is trying to get a diverse enough set of examples, evaling, etc.
If you have great data with simple input output pairs, you should really give it a shot.
I am thinking to fine-tune it to recognize better my handwriting. It already works quite well by default, but my writing is just horrible, so it got trouble sometimes.
