Neural Networks and LLMs notebook with LITE_MODE=False (items_full dataset), the notebook fails during tensor creation.

Code:

X_train_tensor = torch.FloatTensor(X.toarray())

Error:

RuntimeError: [enforce fail at alloc_cpu.cpp:117] data. DefaultCPUAllocator: not enough memory: you tried to allocate 16000000000 bytes.

The issue appears to be caused by converting the sparse matrix produced by HashingVectorizer into a dense array using X.toarray(), which requires approximately 16 GB of RAM for the full dataset (800,000 samples and 5,000 features).

Expected behavior: - The notebook should either run successfully with the full dataset, - Provide guidance about the memory requirements, - Or use a more memory-efficient approach that avoids converting the entire sparse matrix to a dense array.

Environment: - Dataset: items_full - HashingVectorizer(n_features=5000) - PyTorch - Local machine