I/O Modules

TCGA Builders

The case-level builders in pymaftools.io.tcga are the canonical TCGA API. They resolve downloaded GDC files to cases, select the requested sample type, and return the appropriate table class with aligned metadata. The standalone functions in pymaftools.io.tcga_readers remain available as lower-level compatibility utilities.

TCGA-specific data builders and GDC client.

class pymaftools.io.tcga.GDCClient(token_path=None, data_types=None, projects=None, gdc_client_path=None, threads=8, retries=5)[source]

Bases: object

Client for querying, aligning, and downloading TCGA data from GDC.

Parameters:

token_path (str or Path, optional) – GDC authentication token (required for controlled-access data).
data_types (dict, optional) – Data type configurations keyed by label. Each entry must have data_type (GDC field) and optionally workflow_type. Defaults to DATA_TYPE_CONFIGS.
projects (list of str, optional) – GDC project IDs (e.g., ["TCGA-LUAD", "TCGA-LUSC"]).
gdc_client_path (str or Path, optional) – Path to the gdc-client binary. Auto-detected from PATH if not set.
threads (int) – Download threads passed to gdc-client (default 8).
retries (int) – Retry attempts for gdc-client (default 5).

Examples

>>> # From config file (recommended)
>>> client = GDCClient.from_config("config.toml")
>>> client.generate_full_manifests(outdir="data/manifests/full")
>>> client.align_manifests(outdir="data/manifests/aligned")

>>> # Programmatic
>>> client = GDCClient(projects=["TCGA-LUAD"])
>>> aligned = client.align_cases("TCGA-LUAD")

classmethod from_config(config_path='config.toml')[source]

Create a GDCClient from a TOML config file.

Parameters:

config_path (str or Path) –

Path to a TOML file with the following structure:

projects = ["TCGA-LUAD", "TCGA-LUSC"]

[download]
gdc_client = "tools/gdc-client"
token      = "/path/to/gdc-token.txt"
threads    = 8
retries    = 5

[data_types.expression]
data_type     = "Gene Expression Quantification"
workflow_type = "STAR - Counts"

[data_types.mutation]
data_type     = "Masked Somatic Mutation"
workflow_type = "Aliquot Ensemble Somatic Variant Merging and Masking"

Each [data_types.<label>] entry must have data_type and optionally workflow_type. Labels become manifest filename prefixes and raw download directory names.

Return type:

GDCClient

Returns:

GDCClient

get_cases(project_id, data_type_key)[source]

Get all case submitter_ids that have a given data type.

Return type:: set[str]

align_cases(project_id, data_types=None)[source]

Find cases with all specified data types (set intersection).

Return type:: list[str]

align_multi_project(project_ids=None, data_types=None)[source]

Align cases across multiple projects.

Return type:: dict[str, list[str]]

generate_full_manifests(projects=None, outdir='data/manifests/full', mapping_path='data/file_to_case.tsv')[source]

Generate complete per-dtype manifests (no case filtering) and build a file_id → case_id / sample_type / project mapping table.

Parameters:

projects (list of str, optional) – Project IDs. Defaults to self.projects.
outdir (str or Path) – Output directory for full manifests.
mapping_path (str or Path) – Output path for file_to_case.tsv.

Return type:

Path

Returns:

Path – Output manifest directory.

build_file_mapping(records)[source]

Build file_id → case_id / sample_type / project mapping via GDC API.

Parameters:: records (list of dict) – Each dict must have file_id, filename, dtype.
Return type:: DataFrame
Returns:: pd.DataFrame – Columns: file_id, filename, dtype, case_id, sample_type, project.

align_manifests(mode='pipeline', full_manifest_dir='data/manifests/full', portal_path=None, mapping_path='data/file_to_case.tsv', outdir='data/manifests/aligned', aligned_cases_path='data/aligned_cases.tsv')[source]

Align manifests across data types — keep only cases with all omics.

Parameters:

mode (str) – "pipeline" (read full manifests + file_to_case.tsv) or "portal" (read a GDC portal manifest, query GDC for metadata).
full_manifest_dir (str or Path) – Directory containing full per-dtype manifests (pipeline mode).
portal_path (str or Path, optional) – Path to a GDC portal manifest file (portal mode).
mapping_path (str or Path) – Path to file_to_case.tsv (pipeline mode).
outdir (str or Path) – Output directory for aligned manifests.
aligned_cases_path (str or Path) – Output path for aligned_cases.tsv.

Return type:

Path

Returns:

Path – Aligned manifest directory.

fetch_clinical_table(case_ids)[source]

Fetch structured clinical data from GDC Cases API.

Parameters:: case_ids (list of str) – Case submitter_ids.
Return type:: DataFrame
Returns:: pd.DataFrame – Clinical table indexed by case_id.

get_file_metadata(project_id, data_type_key, case_ids=None)[source]

Get file metadata for a project + data type, optionally filtered by cases.

Return type:: DataFrame

download(case_ids, project_id, data_types=None, outdir='data/raw', manifest_dir=None)[source]

Generate manifests for aligned cases and download via gdc-client.

Parameters:

case_ids (list of str) – Aligned case submitter_ids.
project_id (str) – GDC project ID.
data_types (list of str, optional) – Data type keys. Defaults to all.
outdir (str or Path) – Output base directory for downloaded files.
manifest_dir (str or Path, optional) – Where to write intermediate manifests. Defaults to {outdir}/manifests.

Return type:

Path

generate_manifests(case_ids, project_id, data_types=None, outdir='manifests')[source]

Generate GDC download manifests for specific aligned cases.

Return type:: dict[str, Path]

pymaftools.io.tcga.parse_tcga_barcode(barcode)[source]

Parse a TCGA barcode into its components.

Parameters:: barcode (str) – TCGA barcode (e.g., "TCGA-44-2655-01A-01D-0182-01").
Return type:: dict
Returns:: dict – Parsed components: project, tss, participant, case_id, and optionally sample_type, vial, portion, analyte, plate, center.

class pymaftools.io.tcga.TCGATableBuilder(data_dir, mapping, sample_type='Primary Tumor')[source]

Bases: ABC

Base builder for TCGA data types.

Subclasses must implement: - file_pattern: glob pattern for matching files - read_and_merge(files): read files and return a table

Parameters:

data_dir (str or Path) – Directory containing downloaded files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.
sample_type (str or None, default "Primary Tumor") – Sample type to retain when building a case-level matrix. Set to None only when the input contains at most one file per case.

file_pattern: str

resolve_files()[source]

Scan directory and resolve files to case metadata.

Return type:: list[dict]

select_files(files)[source]

Select one deterministic input file for each case.

Case-level matrices cannot represent multiple sample types or aliquots under the same column identifier. Filtering happens before both matrix construction and metadata construction so that they cannot disagree.

Return type:: list[dict]

abstractmethod read_and_merge(files)[source]

Read resolved files and merge into a table.

Parameters:: files (list of dict) – Output from resolve_files().
Returns:: PivotTable or subclass – Table with columns as sample identifiers.

build_sample_metadata(table, files)[source]

Build sample_metadata from resolved file info.

Creates a DataFrame indexed by case_id with columns: case_id, sample_type, file_id, data_type.

Parameters:

table (PivotTable) – The built table (used to get column names).
files (list of dict) – Resolved file info.

Return type:

DataFrame

Returns:

pd.DataFrame – Sample metadata indexed by case_id.

build()[source]

Execute the full build pipeline.

Returns:: PivotTable or subclass – Complete table with sample_metadata attached.

class pymaftools.io.tcga.TCGAExpressionBuilder(data_dir, mapping, count_column='unstranded', enrich_coordinates=True, sample_type='Primary Tumor')[source]

Bases: TCGATableBuilder

Build an ExpressionTable from TCGA STAR-Counts files.

Parameters:

data_dir (str or Path) – Directory containing expression files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.
count_column (str, default "unstranded") – Column to extract from STAR-Counts files. Options: unstranded, stranded_first, stranded_second, tpm_unstranded, fpkm_unstranded, fpkm_uq_unstranded.
enrich_coordinates (bool, default True) – If True, enrich feature_metadata with genomic coordinates (chromosome_name, start_position, end_position) from Ensembl BioMart via pymaftools.utils.geneinfo.load_ensembl_map(). The first call downloads and caches the map; subsequent calls are fast.
sample_type (str or None, default "Primary Tumor") – Sample type to retain in the case-level matrix.

file_pattern: str = '*.rna_seq.augmented_star_gene_counts.tsv'

read_and_merge(files)[source]

Read resolved files and merge into a table.

Parameters:: files (list of dict) – Output from resolve_files().
Return type:: ExpressionTable
Returns:: PivotTable or subclass – Table with columns as sample identifiers.

class pymaftools.io.tcga.TCGAMutationBuilder(data_dir, mapping, sample_type='Primary Tumor')[source]

Bases: TCGATableBuilder

Build a MAF object from TCGA masked somatic mutation files.

Note: MAF is a flat DataFrame (not a PivotTable matrix). Use maf.to_pivot_table() to get a gene × sample matrix.

Parameters:

data_dir (str or Path) – Directory containing .maf.gz files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.

file_pattern: str = '*.maf.gz'

read_and_merge(files, tumor_only=True)[source]

Read resolved files and merge into a table.

Parameters:: files (list of dict) – Output from resolve_files().
Return type:: MAF
Returns:: PivotTable or subclass – Table with columns as sample identifiers.

build_sample_metadata(table, files)[source]

Build sample_metadata from resolved file info.

Creates a DataFrame indexed by case_id with columns: case_id, sample_type, file_id, data_type.

Parameters:

table (PivotTable) – The built table (used to get column names).
files (list of dict) – Resolved file info.

Returns:

pd.DataFrame – Sample metadata indexed by case_id.

build()[source]

Execute the full build pipeline.

Return type:: MAF
Returns:: PivotTable or subclass – Complete table with sample_metadata attached.

class pymaftools.io.tcga.TCGACNVSegmentBuilder(data_dir, mapping, sample_type='Primary Tumor')[source]

Bases: TCGATableBuilder

Build CNV data from TCGA masked segment files.

Provides two outputs: - build() → raw segment DataFrame (long format) - build_cytoband_table() → cytoband × sample CopyNumberVariationTable

Parameters:

data_dir (str or Path) – Directory containing .seg.v2.txt files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.

file_pattern: str = '*.ascat3.allelic_specific.seg.txt'

read_and_merge(files)[source]

Read resolved files and merge into a table.

Parameters:: files (list of dict) – Output from resolve_files().
Return type:: DataFrame
Returns:: PivotTable or subclass – Table with columns as sample identifiers.

build()[source]

Build raw segment DataFrame (long format).

Return type:: DataFrame

build_cytoband_table(seg_df=None, cytoband_path=None)[source]

Convert segments to cytoband × sample matrix.

Computes overlap-weighted average Segment_Mean per cytoband per case.

Parameters:

seg_df (pd.DataFrame, optional) – Pre-built segment DataFrame. If None, calls build() first.
cytoband_path (str or Path, optional) – Path to cytoBand.txt. Defaults to bundled file.

Return type:

CopyNumberVariationTable

Returns:

CopyNumberVariationTable – Cytoband × sample matrix.

class pymaftools.io.tcga.TCGACNVGeneBuilder(data_dir, mapping, value_column='copy_number', strip_gene_version=True)[source]

Bases: TCGATableBuilder

Build a CopyNumberVariationTable from TCGA gene-level copy number files.

Supports both ASCAT3 filename variants: - *.gene_level_copy_number.v36.tsv - *.gene_level.copy_number_variation.tsv

Parameters:

data_dir (str or Path) – Directory containing gene-level CNV files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.
value_column (str, default "copy_number") – Column to use as values. Options: copy_number, min_copy_number, max_copy_number.
strip_gene_version (bool, default True) – Strip ENSG version suffix (e.g. .15).

file_pattern: str = '*.gene_level_copy_number.v36.tsv'

resolve_files()[source]

Scan directory and resolve files to case metadata.

Return type:: list[dict]

read_and_merge(files)[source]

Read resolved files and merge into a table.

Parameters:: files (list of dict) – Output from resolve_files().
Return type:: CopyNumberVariationTable
Returns:: PivotTable or subclass – Table with columns as sample identifiers.

class pymaftools.io.tcga.TCGAMethylationBuilder(data_dir, mapping, sample_type='Primary Tumor')[source]

Bases: TCGATableBuilder

Build a PivotTable from TCGA methylation beta value files.

Parameters:

data_dir (str or Path) – Directory containing methylation beta files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.

file_pattern: str = '*.methylation_array.sesame.level3betas.txt'

read_and_merge(files)[source]

Read resolved files and merge into a table.

Parameters:: files (list of dict) – Output from resolve_files().
Return type:: PivotTable
Returns:: PivotTable or subclass – Table with columns as sample identifiers.

class pymaftools.io.tcga.TCGAClinicalBuilder(data_dir, mapping, file_type='patient')[source]

Bases: object

Build a clinical DataFrame from TCGA BCR clinical TXT files.

Parameters:

data_dir (str or Path) – Directory containing clinical supplement files.
mapping (str, Path, or pd.DataFrame) – Path to file_to_case.tsv or pre-loaded mapping DataFrame.
file_type (str, default "patient") – Type of clinical file: patient, drug, radiation, follow_up, nte.

build()[source]

Read and merge clinical TXT files.

Return type:: DataFrame
Returns:: pd.DataFrame – Clinical table indexed by bcr_patient_barcode.

pymaftools.io.tcga.load_file_mapping(mapping_path)[source]

Load file_to_case.tsv mapping table.

Expected columns: file_id, filename, data_type, case_id, sample_type.

Parameters:: mapping_path (str or Path) – Path to the TSV mapping file.
Return type:: DataFrame
Returns:: pd.DataFrame – Mapping table indexed by file_id.

pymaftools.io.tcga.resolve_files(data_dir, pattern, mapping_df)[source]

Scan a download directory and resolve each file to its case metadata.

Supports two directory layouts: - gdc-client: {data_dir}/{file_uuid}/{filename} - flat: {data_dir}/{filename}

Parameters:

data_dir (str or Path) – Base directory containing downloaded files.
pattern (str) – Glob pattern for the data files.
mapping_df (pd.DataFrame) – Mapping table from load_file_mapping(), indexed by file_id.

Return type:

list[dict]

Returns:

list of dict – Each dict has keys: case_id, sample_type, data_type, file_id, filepath. Sorted by case_id.

VCF Parsing

pymaftools.io.vcf.parsers.sample_type(sample_id)[source]

Classify a VCF sample column as tumour or normal.

Parameters:: sample_id (str) – Sample identifier from the VCF header.
Return type:: str
Returns:: str – "TUMOR" or "NORMAL".
Raises:: ValueError – If the sample identifier does not end with T or N.

pymaftools.io.vcf.parsers.parse_info(format_str, value_str, *, alt_index=0)[source]

Parse caller-specific FORMAT data into depth metrics.

Parameters:

format_str (str) – FORMAT column value.
value_str (str) – Sample-specific value string matching format_str.
alt_index (int, default 0) – Zero-based alternate allele index for multi-allelic records.

Return type:

tuple[int, int, int, float]

Returns:

tuple[int, int, int, float] – Parsed (DP, ref_count, alt_count, AF) values. ref_count is the reference allele depth.

Raises:

ValueError – If the FORMAT pattern is not recognised or cannot be parsed.

pymaftools.io.vcf.parsers.parse_vcf_rows(vcf_path, caller)[source]

Parse PASS VCF rows into validated dictionaries.

Parameters:

vcf_path (str) – Path to the VCF file.
caller (str) – Variant caller name assigned to each row.

Return type:

list[dict]

Returns:

list[dict] – Parsed and validated VCF rows.

class pymaftools.io.vcf.record.VCFRecord(chrom, pos, ref, alt, filter, caller, tumor_dp, tumor_ref, tumor_ad, tumor_af, normal_dp=None, normal_ref=None, normal_ad=None)[source]

Bases: object

A validated representation of a parsed VCF variant row.

chrom

Chromosome label in chr notation.

Type:: str

pos

1-based genomic position.

Type:: int

ref

Reference allele.

Type:: str

alt

Alternate allele.

Type:: str

filter

VCF FILTER value.

Type:: str

caller

Variant caller name.

Type:: str

tumor_dp

Tumour sample total read depth.

Type:: int

tumor_ad

Tumour sample alternate allele depth.

Type:: int

tumor_af

Tumour sample alternate allele fraction.

Type:: float

normal_dp

Normal sample total read depth.

Type:: int or None, default None

normal_ad

Normal sample alternate allele depth.

Type:: int or None, default None

tumor_ref: int

normal_ref: int | None = None