Open in app

Sign In

Write

Sign In

23andMe Engineering
23andMe Engineering

111 Followers

Home

About

Published in 23andMe Engineering

·Mar 2, 2022

Celery at Scale at 23andMe

Scanner Luce, Technical Lead Engineer at 23andMe The shape of the problem At 23andMe, we maintain customer-facing websites as well as backend compute pipelines that process a staggering amount of data. This requires a system that can asynchronously, as well as on a schedule, execute tasks quickly and reliably. We need to know how reliable…

Celery

6 min read

Celery at Scale at 23andMe
Celery at Scale at 23andMe
Celery

6 min read


Published in 23andMe Engineering

·Dec 20, 2021

Developing safe and reliable ML products at 23andMe

Manoj Ganesan, Software Architect at 23andMe — ML products at 23andMe Data and Machine Learning are core components of 23andMe’s Health + Ancestry and 23andMe+ services. Since 23andMe launched in 2007, over 80% of our approximately 12 million customers have consented to participate in research and contribute their genetic and self-reported survey data to help the advancement of science. …

Machine Learning

12 min read

Developing safe and reliable ML products at 23andMe
Developing safe and reliable ML products at 23andMe
Machine Learning

12 min read


Published in 23andMe Engineering

·Oct 27, 2021

Introduction to Nextflow

Anuved Verma and Anja Bog, Software Engineers at 23andMe — Are you planning to run “some kind of workflow” and don’t want to worry about infrastructure, monitoring, or resumability? Do you want to orchestrate different kinds of tasks, like Python scripts, command-line tools, or applications written in other programming languages? Yes? …

Netflow

17 min read

Introduction to Nextflow
Introduction to Nextflow
Netflow

17 min read


Published in 23andMe Engineering

·Apr 6, 2021

Processing Large Files through Unix pipeline in AWS Lambda Function

Patrick Yee and Allie Sanzi, Software Engineers at 23andMe — Summary This article illustrates how to architect an AWS Lambda function, written in Python, to stream input data from an S3 object, pipe the data stream through an external program, and then pipe the output stream to an object in S3. Introduction AWS Lambda function is a handy serverless computing service for…

AWS

3 min read

Processing Large Files through Unix pipeline in AWS Lambda Function
Processing Large Files through Unix pipeline in AWS Lambda Function
AWS

3 min read


Published in 23andMe Engineering

·Mar 1, 2021

Reducing AWS EMR costs with Spot, Task Nodes, and Instance Types

Prad Pagadala, Software Engineer at 23andMe — AWS Elastic Map Reduce (EMR), a popular big data platform, powers a lot of computation at 23andMe. One particular use case is to find associations between genes and traits using Hail, which is built on top of Apache Spark. In the course of figuring out how to optimize processing for…

AWS

5 min read

Reducing AWS EMR costs with Spot, Task Nodes, and Instance Types
Reducing AWS EMR costs with Spot, Task Nodes, and Instance Types
AWS

5 min read


Published in 23andMe Engineering

·Feb 8, 2021

High-performance genetic datastore on AWS S3 using Parquet and Arrow

Tulasi Paradarami, Sr. Engineering Manager at 23andMe — Introduction In bioinformatics, Variant Call Format (VCF) is a popular text file format for storing genetic variation data, and is the standard output for popular imputation methodologies like minimac. It’s unambiguous, it’s flexible, and it supports arbitrary metadata. A drawback of the VCF format, however, is that it’s a text file…

Parquet

8 min read

High-performance genetic datastore on AWS S3 using Parquet and Arrow
High-performance genetic datastore on AWS S3 using Parquet and Arrow
Parquet

8 min read


Published in 23andMe Engineering

·Updated Oct 26, 2021

Accessibility and Me

Joe Banks, Tech Lead at 23andMe — I have a disability. Two, actually. I was diagnosed with Usher Syndrome Type 2A, a rare genetic disease characterized by hearing loss and progressive vision loss. When I was 3, my parents discovered that I was born with moderate-severe hearing loss in both of my ears. …

Accessibility

9 min read

Accessibility and Me
Accessibility and Me
Accessibility

9 min read

23andMe Engineering

23andMe Engineering

111 Followers

Get your Codon

Following
  • Palantir

    Palantir

  • Pinterest Engineering

    Pinterest Engineering

  • Prototypr Editors

    Prototypr Editors

  • Raphaël Zumer

    Raphaël Zumer

  • Kritika Jain

    Kritika Jain

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech