🚀 Ilum's JupyterLab via VS Code on Local Machine

This guide explains how to connect your local VS Code installation to Ilum's JupyterLab kernel, allowing you to work with Ilum notebooks directly from your preferred development environment while using Ilum's Spark clusters and data infrastructure.

📋 Prerequisites​

Before you begin, ensure you have the following:

• Local VS Code Installation: Download and install VS Code on your machine
• VS Code Extensions:
  • Python extension
  • Jupyter extension
• kubectl Access: Configured access to your Ilum cluster with appropriate permissions
• Ilum JupyterLab Service: The Ilum JupyterLab service should be running in your cluster

🛠️ Methods of connecting to Ilum's JupyterLab​

• Ilum-jupyter service
• VS Code Remote-SSH

🌐 Connect to external Jupyter Server via ilum-jupyter service​

Step 1: Expose the Jupyter Service​

First, you need to expose the Ilum JupyterLab service to your local machine. You have several options:

Option A - Port Forwarding (Recommended):

কুবেক্টল পোর্ট-ফরোয়ার্ড এসভিসি / আইএলইউএম-জুপিটার 8888: 8888 

This command creates a secure tunnel from your local machine's port 8888 to the Ilum JupyterLab service. Keep this terminal session open while working with the notebooks.

Option B - NodePort, LoadBalancer or Ingress: Alternatively, you can expose the ilum-jupyter service using NodePort, LoadBalancer or Ingress for more persistent access.

Using NodePort Service​

kubectl expose deployment ilum-jupyter --type NodePort --port 8888 --name ilum-jupyter-external
kubectl get svc ilum-jupyter-external

Then use the external IP and NodePort in your VS Code connection URL.

Using LoadBalancer​

kubectl expose deployment ilum-jupyter --type LoadBalancer --port 8888 --name ilum-jupyter-external
kubectl get svc ilum-jupyter-external

Use the external LoadBalancer IP in your VS Code connection URL.

Step 2: Create or Open a Notebook File​

In VS Code, create a new file with the .ipynb extension or open an existing Jupyter notebook file.

Step 3: Select Kernel​

Once you have a notebook file open, click on the "Select Kernel" button in the top-right corner of the notebook interface.

Step 4: Choose Existing Jupyter Server​

From the kernel selection menu, select the "Existing Jupyter Server" option.

Step 5: Enter Remote URL​

You'll be prompted to specify the URL of the running Jupyter server. Enter one of the following connection URLs:

Option A - Using Port Forward:

http://localhost:8888/external/jupyter

Option B - Direct Service Access: If you have direct access to the ilum-jupyter service (e.g., via NodePort or LoadBalancer), substitute localhost:8888 with your service address:

http://your-service-address:port/external/jupyter

Step 6: Allow HTTP Connection​

VS Code will show a security warning about connecting over HTTP without a token. Since this is a secure port-forward connection to your trusted Ilum cluster, click "Yes" to allow the HTTP connection.

Step 7: Select the Python Kernel​

Finally, choose the "ipykernel" option from the available kernels. This will be the Python kernel from your Ilum JupyterLab environment.

💻 Working with Ilum Notebooks​

Once connected, you can write and execute notebook cells just as you would in Ilum's JupyterLab interface. The code runs on the Ilum cluster, giving you access to:

• Spark Clusters: Use Sparkmagic commands like %manage_spark এবং %%sস্পার্ক
• Data Storage: Access your Ilum data volumes and S3 storage
• Libraries: All pre-installed libraries and packages from the Ilum environment
• সংস্থান : Computational resources allocated to your Ilum workspace

Example: Using Spark in VS Code​

# Load sparkmagic extension
% load_ext sparkmagic. magics

# Start a Spark session using Sparkmagic
% manage_spark

# Run Spark code
% % স্ফুলিঙ্গ 
আরডিডি = এস.সি. . সমান্তরাল ( [ 1 ,  2 ,  3 ,  4 ,  5 ,  6 ,  7 ,  8 ,  9 ,  10 ] ) 
    মুদ্রণ ( আরডিডি . গণনা ( ) ) 

📖 Working with IlumIntro Notebooks​

You can access and work with the IlumIntro tutorial notebooks that are available in your Ilum JupyterLab environment. These notebooks provide guided examples and tutorials for using Ilum's features.

To work with IlumIntro notebooks:

1. Access from JupyterLab: First, open the IlumIntro notebooks in your Ilum JupyterLab interface
2. Export to Local: Download the notebook files (.ipynb) to your local machine
3. Open in VS Code: Open these files in VS Code using the connection established above
4. Work Locally: Edit and run the notebooks in VS Code while maintaining the connection to Ilum's compute resources

For more information about IlumIntro notebooks and tutorials, see the Notebook Usage Guide.

🔄 Import/Export Workflow​

Exporting from JupyterLab to VS Code​

1. In Ilum JupyterLab, right-click on the notebook file
2. Select "Download" to save the .ipynb file to your local machine
3. Open the downloaded file in VS Code
4. Follow the connection steps above to link it to your Ilum kernel

Importing from VS Code to JupyterLab​

1. Save your notebook in VS Code (Ctrl+S / Cmd+S)
2. Upload the .ipynb file to your Ilum JupyterLab workspace:
   • Use the file browser in JupyterLab
   • Click the upload button and select your notebook file
   • Alternatively, use Git if your workspace is connected to version control

🛠️ Troubleshooting​

Connection Issues​

• Port already in use: If port 8888 is busy, use a different local port:

  kubectl port-forward svc/ilum-jupyter 8889:8888
  

  Then use http://localhost:8889/external/jupyter in VS Code.

• পরিষেবা খুঁজে পাওয়া যায়নি : Verify the correct service name and namespace:

  kubectl get svc | grep jupyter
  

Kernel Issues​

• Kernel not starting: Ensure the port-forward connection is active and the URL is correct
• Authentication errors: Verify that your kubectl context has the necessary permissions
• Library not found: Remember that code executes on the Ilum cluster, so only libraries available in the Ilum environment can be used

Performance Considerations​

• Network latency: Code execution happens remotely, so there may be slight delays compared to local execution
• Large outputs: Avoid displaying very large datasets directly; use .head(), .sample(), or similar methods
• Session management: Close unused kernels to free up cluster resources

🔐 Security Notes​

• All code execution happens within your Ilum cluster's security boundaries
• No sensitive data is transmitted outside your cluster environment
• Always use trusted networks when establishing these connections

🔗 VS Code Remote-SSH​

This section provides a template for connecting to Ilum's Jupyter pod directly via VS Code's Remote-SSH extension. This approach allows you to work directly within the pod environment rather than connecting to the Jupyter server via HTTP.

📋 Prerequisites for Remote-SSH Connection​

• VS Code Remote-SSH Extension: Install the Remote-SSH extension

Step 1: Enable SSH mode in jupyter deployment​

Firstly we need to generate secret with SSH keys and enable SSH access in the Jupyter deployment. You can accomplish this by upgrading the Helm release with SSH enabled.

Generate SSH keys if you don't have them already:

mkdir -p jupyter-ssh-keys
cd jupyter-ssh-keys

# Generate RSA, ECDSA, and ED25519 host keys
ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N ""
ssh-keygen -t ecdsa -b 521 -f ssh_host_ecdsa_key -N ""
ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N ""

# Optional: create authorized_keys for jovyan user
ssh-keygen -t ed25519 -C "ilum-jupyter" -f jovyan_id_ed25519 -N ""
cp jovyan_id_ed25519.pub authorized_keys

This command will create all keys needed for SSH. Now in jupyter-ssh-keys you have:

• ssh_host_rsa_key
• ssh_host_rsa_key.pub
• ssh_host_ecdsa_key
• ssh_host_ecdsa_key.pub
• ssh_host_ed25519_key
• ssh_host_ed25519_key.pub
• authorized_keys
• jovyan_id_ed25519
• jovyan_id_ed25519.pub

Now we have to create kubernetes secret with these keys:

kubectl create secret generic ilum-jupyter-ssh-keys \
  --from-file=ssh_host_rsa_key=ssh_host_rsa_key \
  --from-file=ssh_host_rsa_key.pub=ssh_host_rsa_key.pub \
  --from-file=ssh_host_ecdsa_key=ssh_host_ecdsa_key \
  --from-file=ssh_host_ecdsa_key.pub=ssh_host_ecdsa_key.pub \
  --from-file=ssh_host_ed25519_key=ssh_host_ed25519_key \
  --from-file=ssh_host_ed25519_key.pub=ssh_host_ed25519_key.pub \
  --from-file=authorized_keys=authorized_keys

Then upgrade the Helm release with SSH enabled:

হেলম আপগ্রেড ইলুম ইলুম / ইলুম \ 
    --set ilum-jupyter.ssh.enabled=true \
    --set resources.limits.memory=2Gi \
    --set resources.requests.memory=2Gi \
--পুনঃব্যবহার-মান 

Step 2: Configure SSH Connection​

In file ~/.ssh/config, add a new host configuration for the Jupyter pod. By default, ilum-jupyter in ssh mode exposes ssh with nodePort service, but you can also utilize port-forwarding to access the pod. Here is an exemplary configuration. Fill the HostName, ব্যবহারকারী , Portএবং IdentityFile fields according to your setup:

Fetch service:

kubectl get svc ilum-jupyter-ssh

ilum-jupyter-ssh                NodePort    10.110.84.63     <none>        2222:31309/TCP      63s

and use the node ipএবং PORT(S) values to fill in the HostNameএবং Port fields.

Add to your SSH config (~/.ssh/config):

Host ilum-jupyter-pod
    HostName node_address
    User jovyan
    Port 31309
    IdentityFile (path to directory created above/jovyan_id_ed25519, eg. ~/jupyter-ssh-keys/jovyan_id_ed25519)
    IdentitiesOnly yes

Step 3: Connect via VS Code​

1. Open VS Code Command Palette (Ctrl+Shift+P / Cmd+Shift+P)
2. Type "Remote-SSH: Connect to Host"
3. Select your configured host: ilum-jupyter-pod
4. VS Code will open a new window connected to the pod environment

Step 4: Configure VS Code for Jupyter​

1. Continue on fingerprint information

2. Open folder /home/jovyan in the remote window

3. Install VS Code Extensions in SSH remote window (Python, Jupyter)

4. Select the Python interpreter in the remote window (Python 3)

5. With kernel selected, you can now create or open .ipynb files and run code directly in the Jupyter pod environment.

6. You can monitor all of your workloads in the JupyterLab interface as well. Spark session started from VS Code will appear in the ILUM UI.

✨ Benefits of Remote-SSH Method​

• Direct pod access: Work directly within the Jupyter pod environment
• Full filesystem access: Access all files and directories in the pod
• Native performance: No network latency for file operations
• Development environment: Full access to installed packages and tools in the pod

🤔 Considerations​

• Network configuration: Requires proper SSH access and network routing to the cluster
• নিরাপত্তা : Ensure SSH access follows your organization's security policies
• Pod lifecycle: Connection will be lost if the pod restarts or gets rescheduled
• Resource sharing: Multiple SSH connections share the same pod resources

Note: The specific implementation details will vary based on your cluster configuration, network setup, and security requirements. This template should be customized according to your environment.

🛠️ Troubleshooting​

Connection Issues - disconnections​

• SSH connection drops: Ensure jupyter pod has adequate resources as ssh might be resource consuming, you can always manipulate resources in the Helm chart values and upgrade the release.
• Firewall issues: Ensure that your local machine can reach the cluster nodes on the specified SSH port
• Incorrect SSH config: Double-check the ~/.ssh/config entries for typos or incorrect values

Kernel Issues​

• Kernel not starting: Ensure the SSH connection is active and the Python interpreter is correctly set
• Extensions not working: Make sure the required VS Code extensions are installed in the remote environment, in case of jupyter pod restart you might need to reinstall them
• Library not found: Remember that code executes in the Jupyter pod, so only libraries available in that environment can be used

Performance Considerations​

• Network latency: While file operations are local to the pod, any external network calls will still be subject to latency
• Resource limits: Be aware of the resource limits of the Jupyter pod, as multiple users or heavy workloads can impact performance
• Session management: Close unused connections to free up pod resources

🔐 Security Notes​

• All code execution happens within your Ilum cluster's security boundaries
• No sensitive data is transmitted outside your cluster environment
• Always use trusted networks when establishing these connections

🚀 Next Steps​

Once you have VS Code successfully connected to your Ilum JupyterLab environment, explore:

• JupyterLab Features
• Spark Integration Guide
• Data Science Platform Features

💬 Support​

For additional help or questions about VS Code integration with Ilum, contact us at [email protected] or refer to our Help documentation.