NVidia GeForce RTX 50x0 cards - 70% performance increase, but AI > you

Cinge said:

I'm honestly waiting until they just say fuck it and nuke everything but the flagship card and let amd/intel have all the stuff below that. Unless for some reason their AI/Datacenter business crashes.

Click to expand...

That probably won't happen, because Jensens ego won't allow to admit defeat in a field he once dominated. That man probably hasn't heard the word "no" in years.

Also, especially after DeepSeek we are approaching the "trough of disillusionment" in regards to "AI"--especially when people realize how it fundamentally works, and why it's not going to get much better, regardless how much money you throw at it.

I can explain this in detail, but it will probably a long rant, and I don't want to type it out if no one's interested.

Neranja said:

That probably won't happen, because Jensens ego won't allow to admit defeat in a field he once dominated. That man probably hasn't heard the word "no" in years.

Also, especially after DeepSeek we are approaching the "trough of disillusionment" in regards to "AI"--especially when people realize how it fundamentally works, and why it's not going to get much better, regardless how much money you throw at it.

I can explain this in detail, but it will probably a long rant, and I don't want to type it out if no one's interested.

Click to expand...

Go ahead

Hateyou said:

Go ahead

Click to expand...

This is an ELI5 version of how AI works: If you enter a prompt, the "AI" -- or more precise, the "language model" responds with a chain of words. The AI puts these together according to the statistical probability of what the next word according to a) the input and b) the training data would be (probability vector, or stochastic vector).

The AI bros claim that to improve the quality of the output and make the model more "intelligent", we just need to build a bigger model, with more parameters. Which requires more memory and computational power:



This is why OpenAI wants $500 billion: They dangle the "singularity" (when "AI is becoming smarter than humans" and improves itself) at the end of the curve above, like a carrot in front of a donkey. "Trust us, we just need a bigger boat."

The problem with that assumption is: The curve goes probably more like this:



And here's a video you can now watch:

First of all, let me preface this by stating that I didn't claim AI models were useless, just that we are approaching the "trough of disillusionment". In fact, we already made very useful things 20 years ago with similar statistical models, like email SPAM filters using Naive Bayes classifiers and Markov networks.

How do you think SPAM filters work? They get trained. Still, the accuracy for such a seemingly simple task compared to full-blown LLMs never got to 100%.

Tripamang said:

Old news bro, thinking models have proven the wall is a lie and the more compute you give the model to spend on a question at inference time the more accurate the response.

Click to expand...

So this is still the current cope? Just spend more time on compute? You realize we are quickly approaching the end of Moore's Law, as improvements in computational power seem to be tied more to node shrinks instead of architectural design nowadays (*cough* Blackwell *cough*), but there is a physical limit on how much further we can shrink? Production is getting more complex and expensive as well, killing yields?

Ponder about this for a while: 3 nm is around 27 silicon atoms.

No, ponder about it some more. Now after some pondering, answer this: How much more do you expect to shrink before yields fall off a cliff, single atom defects make a trace and therefore the complete die unusable, or even worse: the field effects for transistors just outright stop working?

Tripamang said:

In fact the models are likely too big as they are now and we could spend more time training them on logic and less on sheer data as it makes them more accurate.

Click to expand...

Funny enough, this is what DeepSeek innovated: They have a bag that contains multiple models that they call Mixture-of-Experts.

OpenAI's first response ("they stole our training data/models") was very telling.

Tripamang said:

If they are smart enough to understand a topic but not necessarily trained on it they can give better answers by researching the topics before answering then creating the output. You're pushing the compute from training to inference time, and getting better responses.

Click to expand...

And here we have Ouroboros eating its tail. What you describe, in layman terms, is just "let me google that for you, and also build a model real quick".

a) Building models is a pretty expensive operation
b) Search engines themselves are starting to use AI to rank and filter results, feeding the model tainted data, so you need to build your own Google for untainted data
c) But the internet is for porn, so 80% of it is trite shit. Even worse, now more and more of shit is AI generated, which feeds into the models.
d) Everyone is starting to get realy pissed about AI bros collecting as much data as possible, with hammering servers, ignoring robots.txt, and outright pirating 82 TB of books.

Expect a lot more laws and legal precedents to fall into place to regulate AI, now that everyone realizes the AI bros behaved like a bull in a china shop to acquire "training data", and the "it's just training data" excuse stops working because people realize the underlying statistical nature of AI.


There is an even more fundamental challenge in regards to AI: it gets trained on "human knowledge." We can expect--from experience--that at any point in time, 50% of human knowledge is probably to most likely wrong.
Do you still believe the geocentric model, or that smoking doesn't cause cancer? Did you have a lobotomy yet? It's an easy procedure, you know:It won the 1949 Nobel Prize for physiology or medicine. No? Why don't you trust the science?

The current state of human knowledge is fucked, because we incentivized "publish or perish", which lead to loads of junk papers and the cherry-picking or even outright falsifying of data, and are now facing a replication crisis, even in fields that should--by definition of being a STEM field, or by ethical standards--be very much immunie to this, like medicine. If you have time you can go down some deep Chinese rabbit holes.

People who go around clamoring "trust the science" have no idea how science works. And you want to train AI models on all of this?