Видео

Hi Sebastian. I guess this was recorded before 'flatten' was adopted by Pytorch?

would be great to add a video explaining the vanishing/exploding gradient problem in detail! thanks so much!

@16:37 when you say Llama was trained on 1T token, do you still mean there was 32K unique token ? because on your blog post you have "They also have a surprisingly large 151,642 token vocabulary (for reference, Llama 2 uses a 32k vocabulary, and Llama 3.1 uses a 128k token vocabulary); as a rule of thumb, increasing the vocab size by 2x reduces the number of input tokens by 2x so the LLM can fit more tokens into the same input. Also it especially helps with multilingual data and coding to cover words outside the standard English vocabulary."

Your video was incredibly clear and engaging! Thank you for the awesome explanation!

Thanks for explaining

good

watched all 3 vids came back to leave a comment. better explained than all those 1 hour videos

Nice Video. Does LLM mean only auto-regressive models (Not Bert)?

Danke!

You are the best! Thanks a lot for sharing your knowledge to the world.

Dear Dr. Sebastian, please kindly advice. @5:40, you mentioned that the first feature map has 96 channels. So i want to assume that since it is a coloured image, the input must have 3 channels and the total filters based on the first output feature maps = 96/3 =< 32 with the assumptions that the strides = 1 and no padding. Could this also contribute to the success of the architecture, researching and implementing different features in the convolution processes to capture as many features as possible? Could they have succeeded if they used half the feature size? Thank you.

Thank you, Sir. Your lessons are beneficial for the community. Appreciate your hard work..!! 😊

You are a true educator. Honored to be a contributor to one of your libraries.

I know you don't do many tutorials but personally I love theme especially from you!

00:03 PyTorch buffers are essential for implementing large models 01:39 Instantiating a new causal attention without buffers 03:12 Transferring data to GPU using PyTorch Cuda 04:56 Optimizing memory usage during forward pass 06:36 Explanation of creating mask for efficiency in PyTorch Buffers 08:07 Parameters are automatically transferred to GPU, but torch tensors need to be made parameters to be transferred. 10:05 The mask is made a buffer so it's not learned by the optimizer. 11:50 PyTorch buffers facilitate easy transfer of parameters between GPU and CPU

00:02 Three common ways of using large language models 02:39 Developing LLM involves building, pre-training, and fine-tuning. 07:11 LLM predicts the next token in the text 09:30 Training LLM involves sliding fixed size inputs over text data to create batches. 14:22 Byte pair encoding and sentence piece variations allow LLMs to handle unknown words 16:42 Training sets are increasing in size 21:09 Developing an LM involves architecture, pre-training, model evaluation, and fine-tuning. 23:14 The Transformer block is repeated multiple times in the architecture. 27:22 Pre-training creates the Foundation model for fine-tuning 29:28 Training LLMs typically done for one to two epochs 33:44 Pre-training is not usually necessary for adapting LLM for a certain task 35:51 Replace the output layer for efficient classification. 39:54 Classification fine-tuning is key for practical business tasks. 42:01 LLM instruction data set and preference tuning 45:58 Evaluating LLMs is crucial, with MML being a popular metric. 48:07 Multiple choice questions are not sufficient to measure an LM's performance 52:34 Comparing LLM models for performance evaluation 54:32 Continued pre-training is effective for instilling new knowledge in LLMs 58:28 Access slides on the website for more details

One of the best 60 minutes of my time. Really thankful for this..

This is still great stuff, still learning

Thanks Sebastian! I got what is buffer for. Great lecture.

If I am getting this right, for model evaluation using CV, it is done on the whole dataset and then average the model performance, right? Anyways, your videos are very good to the point that I bought your book "Machine Learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python" , even though I use R for my analysis 🤣.

This is such a great explanation with a clear explanation of each step of the algorithm with examples. Thank you!!

Hi, Sebastian I really respect what you are doing. I like your github repository - there are a lot of helpful tutorials I'm going to buy your next book - Build a Large Language Model (From Scratch). i have one question. What minimal gpu do you recommend to have to explore and do all examples from your next book?

Actually learned something new. Thanks Sebastian!

Thank you very much for this explanation.

Great content...thanks a lot

Cheers, great video. I'd suggest being slightly more concise. Either way, great video.

Great Work! I like your LLM notebooks as well!

Thank you for the details. You haven't used the validation set.

Another advantage is that the buffer gets saved in the state_dict when saving the model

It's indeed a clean way to do things but Can't we do the same-thing by adding them as parameter and setting up .requires_grad = False ?

Awesome tutorial🔥

Back to basics. Love it. ❤

The man is back more videos please ❤

I always see this register buffer code in transformer network and never though of the reason would be so simple. Thanks for explaining such ignored concept of pytprch.

I recently purchased llm from scratch from Manning. Amazing learning experience till now

More videos please

Very useful tip 💪💪

I really wish people would stop putting their x link and start sharing something like mastadon or threads, as a free user, x is where u go to feel second class citizen.

Thank you

best explanation ever.

Coming to this video 3years late, now with LLMs being the hot thing around, just wondering if CNN are still the standard for vision or are companies now using transformers for vision?

God Level Explanation From Scratch

In the 4th tip, it was advised to train them separately. But you just calculated the loss separately but that loss was fed as a whole to the train the parameters. Shouldn't the training be done separately as well? first with the real images loss and then on the fake images loss?

Thank you

Thank You Sir 🙏🙏

Can you cite the formula in the end? around 09:15?

Can anyone explain why the deriv is 2*(y-yhat). I can not understand anything after it.

What wonderful Tech Minds : { Sebastian Raschka, Yann LeCun, Andrej Karpathy, ...} who share their works and beautiful ideations for Mere mortal like me... Sebastian's teachings are so, so fundamental that takes fear off my clogged mind... 🙏 Although I am struggling to build LLMs for specific & niche areas, I am confidant of cracking them with great resources like : Build a Large Language Model (From Scratch)!!!

I am your fan, I have most of your books, thanks for this excellent video ! Another evaluation metric that I found interesting in another channel was to make the LLMs to play chess against each other 10 times.

Very good explanation.